Course syllabus

Course-PM

TIF360 / FYM360 Advanced machine learning with neural networks lp4 VT21 (7.5 hp)

Course is offered by the department of Physics

Contact details

Examiner

Giovanni Volpe giovanni.volpe@physics.gu.se

Lecturers

Mats Granath Mats.Granath@physics.gu.se
Kristian Gustafsson Kristian.Gustafsson@physics.gu.se
Bernhard Mehlig Bernhard.Mehlig@physics.gu.se
Daniel Midtvedt daniel.midtvedt@physics.gu.se

Teaching Assistants

Homework A, Graph neural networks: David Fitzek, davidfi@chalmers.se
Homework B, Playing Tetris using Reinforcement Learning: Henrik Moberg, henrik.kleinmoberg@chalmers.se
Homework C: Video frame prediction using transformers: Jesus Pineda, jesus.pineda@physics.gu.se
Homework D: Non-linear time-series analysis with reservoir computers: Ludvig Storm, ludvig.storm@physics.gu.se

Project Groups 1, 3, 5, 7, 9, 11: Henrik Moberg, henrik.kleinmoberg@chalmers.se
Project Groups 2, 4, 6, 8, 10, 12:Harshith Bachimanchi, harshith.bachimanchi@physics.gu.se
Project Groups 13, 15, 17, 19, 21, 23: Yu-Wei Chang, yu-wei.chang@physics.gu.se
Project Groups 14, 16, 18, 20, 22, 24: Jesus Pineda, jesus.pineda@physics.gu.se

Review groups: 1-2, 3-4, ...

Course purpose

This course introduces students to recent developments and state-of-the-art methods in machine learning using artificial neural networks. This advanced course builds on Machine learning with neural networks (FFR135) and provides an in-depth analysis of many of the concepts and algorithms that were briefly introduced in that course, with particular emphasis on applications in the natural and engineering sciences. The goal is to become familiar with several advanced machine-learning methods, and to code them efficiently in Python using current neural-network packages. An essential part of the course are projects in deep learning, recurrent learning, and reinforcement learning.

Schedule

TimeEdit

Course literature

There is no single book that covers all the topics in the course so we have no required books. However, we provide a set of compulsory (and additional) readings assigned to each lecture that covers and perhaps broadens the content of the lectures.

Course design

IMPORTANT: All classes will take place online via ZOOM. The link to the ZOOM meeting for all classes and homeworks is:
https://gu-se.zoom.us/j/6632062637?pwd=U0RJV3FvSmpxdVN4VkpUQ0FlUkgxZz09
Meeting ID: 663 206 2637
Passcode: 488714

Date, time & place	Topic	Readings / Materials to prepare	Additional readings
Tue Mar 23 8:00-9:45	Lecture 1 Giovanni Volpe Introduction Course information
Wed Mar 24 8:00-9:45	Lecture 2 Mats Granath Introduction to Convolutional graph neural networks	Two nice YouTube videos on the topic: Theoretical Foundations of Graph Neural Networks Deep learning on graphs: successes, challenges, and next steps \| Graph Neural Networks Mats lecture notes.	Two papers that give an overview of the subject: A comprehensive survey on graph neural networks. Benchmarkinggraph neural networks Application to drug discovery: A Deep Learning Approach to Antibiotic Discovery.pdf For more in-depth: GRL_Book.pdf
Fri Mar 26 8:00-9:45	Lecture 3 Kristian Gustavsson Introduction to Q-learning and Deep Q-networks	Mehlig, Introduction to machine learning with neural networks (Sections 11.0 and 11.3) Sutton & Barto, Reinforcement Learning, An Introduction (Section 16.5)	Mnih et al., Human- level control through deep reinforcement learning. Nature 518, 529–533 (2015)
Fri Mar 26 10:00-11:45	HW help session
Tue Mar 30 8:00-9:45	Lecture 4 Daniel Midtvedt From convolutional neural networks to transformers	Original paper introducing transformer: https://arxiv.org/pdf/1706.03762.pdf	Youtube video explaining recurrent neural networks: https://youtu.be/LHXXI4-IEns Youtube video explaining transformers and language processing: https://youtu.be/S27pHKBEp30
Wed Mar 31 8:00-9:45	Lecture 5 Bernhard Mehlig Reservoir computers	Mehlig, Introduction to machine learning with neural networks (Section 9.5) Jaeger & Haas, Harnessing nonlinearity, Science 304 (2004) 78-80	Lukosevicius (2012) A practical guide to applying echo state networks Pathak et al. (2017) Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data pathak2017machine.pdf

Wed Mar 31 10:00-11:45	HW help session
Wed Mar 31	Project Group Announced	The project groups will be formed randomly. If, for a very good reasons, this is not optimal for you, inform the instructor before Tue Mar 30.
~Apr 12-30	1st meeting with tutor	Arrange a 45-minute meeting with tutor to discuss and receive feedback on the project idea and outline
Tue Apr 13 8:00-9:45	Lecture 6 (Invited speaker) Luca Biferale Data assimilation and impainting in fluid dynamics	Pathak et al (2016), Context encoders: Feature learning by inpainting. Yeh et al, (2017), Semantic image inpainting with deep generative models Buzzicotti et al (2020), Reconstruction of turbulent data with deep generative models for semantic inpainting from TURB-Rot database	Biferale et al (2020), TURB-Rot. A large database of 3d and 2d snapshots from turbulent rotating flows Di Leoni et al (2020), Synchronization to Big Data: Nudging the Navier-Stokes Equations for Data Assimilation of Turbulent Flows
Fri Apr 16 8:00-11:45	HW correction - session 1
Wed Apr 21 8:00-9:45	Lecture 7 (Invited speaker) Simon Dobnik Learning language and beyond	[1] E. M. Bender and A. Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online, July 2020. Association for Computational Linguistics. https://www.aclweb.org/anthology/2020.acl-main.463 Popular textbooks on natural language processing (read/browse selectively) [2] D. Jurafsky and J. H. Martin. Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition. Third edition draft, Stanford University and University of Colorado at Boulder, December 30 fth. https://web.stanford.edu/~jurafsky/slp3/ [3] J. Eisenstein. Introduction to natural language processing. Adaptive computation and machine learning. The MIT Press: Cambridge, MA and London, England, 2019. https://github.com/jacobeisenstein/gt-nlp-class/tree/master/notes [4] Y. Goldberg. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1):1–309, 2017. https://doi.org/10.2200/S00762ED1V01Y201703HLT037	On ethical considerations E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 610–623, New York, NY, USA, 2021. Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922
Fri Apr 23 8:00-11:45	HW correction - session 2
Tue Apr 27 8:00-9:45	Lecture 8 (Invited speaker) Hampus Linander Model optimization with quantization and pruning	Pruning and Quantization for Deep Neural Network Acceleration: A Survey https://arxiv.org/abs/2101.09671
Fri Apr 30 8:00-11:45	HW correction - session 3
Fri Apr 30 23:59	Project Proposal	Submit your finalize project proposal
~May 1-20	2nd and 3rd meetings with tutor	Arrange 45-minute meetings with tutor to discuss and get feedback on the progress of your project and its presentation
Tue May 4 8:00-9:45	Lecture 9 (Invited speaker) Jonas Einarsson Black-box optimization with Gaussian Processes	Gaussian processes: Chapters 1, 2, 4, 5 (yes, definitely skip Chap. 3) in Rasmussen & Williams: Gaussian Processes for Machine Learning: freely available at http://www.gaussianprocess.org/gpml/ Bayesian optimization: Shahriari et al (2016), Taking the Human Out of the Loop: A Review of Bayesian Optimization. Open access at https://ieeexplore.ieee.org/document/7352306
Tue May 11 10:00-11:45	Lecture 10 (Invited speaker) Daniel Brunner General motivation and how to do experiments with photonic neural networks Towards 20 GHz realtime neural network processors via semiconductor lasers	Tutorial: Photonic neural networks in delay systems D. Brunner, B. Penkovsky, B. A. Marquez, M. Jacquot, I. Fischer, and L. Larger, Journal of Applied Physics 124, 152004 (2018) https://doi.org/10.1063/1.5042342 Guy Van der Sande*, Daniel Brunner and Miguel C. Soriano, Advances in photonic reservoir computing, Nanophotonics 2017; 6(3): 561–576 A complete, parallel and autonomous photonic neural network in a semiconductor multimode laser, Xavier Porte, Anas Skalli, Nasibeh Haghighi, Stephan Reitzenstein, James A. Lott, Daniel Brunner. J. Phys. Photonics in press https://doi.org/10.1088/2515-7647/abf6bd
Tue May 18 8:00-9:45	Lecture 11 (Invited speaker) Carolina Wählby Image analysis and AI in microscopy-based life science research	Deep Learning in Image Cytometry: A Review. Gupta, A. et al., Cytometry Part A 2018 https://onlinelibrary.wiley.com/doi/abs/10.1002/cyto.a.23701
Mon May 24 23:59	Draft of Project Report	Submit a draft of your project report
Wed May 26 23:59	Project Presentation	Submit your project presentation (8-minute video, format MOV)
Fri May 28 8:00-12:00	Project Presentation (Examination)
~May 24-31	4th meeting with tutor	Arrange a 45-minute meeting with tutor to get feedback on your project draft and presentation
Fri Jun 11 23:59	Project Report	Submit your final project report

Learning objectives and syllabus

Learning objectives:

Knowledge and understanding

- Describe the different available neural network models with their advantages and disadvantages

- Find relevant literature to keep up with this quickly advancing field

Skills and ability

- Implement a broad range of state-of-the-art neural network models

- Train and validate these models

- Optimize these models for a specific task

- Plan, manage and execute a small scale project in the field

- Write a report of their results of the project

Judgement and approach

- Critically analyse the advantages and disadvantages of the available neural network models

- Benchmark the results of a neural network models against other models

- Critically evaluate and discuss advances in the field of neural networks

Link to the syllabus on the Chalmers Studieportalen:

Study plan

Link to the syllabus on the GU website:

https://utbildning.gu.se/kurser/kurs_information?courseid=FYM750

Examination form

There will be no written final examination in this course.

The examination consists of homework assignments and a group project.
Specifically, the examination is based on

30% homeworks (10% for each)
20% final project presentation
50% final project report

A necessary (but not sufficient) requirement for passing grade is that 5/10 points are achieved in each homework.

Homework information

The purpose of the homework problem sets is to provide you with hands-on experience of advanced deep learning techniques. You are strongly encouraged to team up and collaborate on the homework, to ask your classmates if you are stuck at some point, and to assist classmates in need of advice. But you must write and run your own code, and have your own work assessed. No late assignments will be accepted.

Grading: Out of the four available homeworks, you need to solve and get assessed on three homeworks.
Each homework accounts for 10 points. A necessary (but not sufficient) requirement for passing grade is that 5/10 points are achieved in each homework.
Overall these three homeworks account for 30% of your total course grade (10% for each).

Computer labs: The computer lab sessions serve two purposes:
1) To get help if you are stuck in a problem set (and have not managed to get unstuck by consulting some of your classmates). The time we have available for computer lab is limited, so it is essential that you come prepared to the lab, with specific questions you need to have answered.
2) To have your homework graded.

Assessment: We will assess your work during lab hours.
Be prepared for the assessment: Generate all your figures and/or videos up front. There will be no time for running code during the assessment. Check that you have answered all questions carefully. You should be able to give reasoned answers regarding your programming choices as well as being able to discuss the implications of your results.

Project information

Group advanced deep learning project

The purpose of the group project work is to provide you with training in

Developing your own small research-style project.
Executing a collaborative project.
Presenting your independent work in writing and as a summary talk.

The deliverables are a 8 minute presentation + 10 minute questions (20% of the course grade) and a written report (50% of the course grade).

The deadline for submitting a project proposal to your tutor (see below) is by 23:59, April 30, 2021. However, it is probably good for you to have it done earlier. Groups with late submission may not receive tutoring.

The draft report is due by 23:59, May 24, 2021.

The project will be presented to the class in week 21 (24-28 May 2021).

The final written report is due by 23:59, June 11, 2021. No late reports will be accepted.

General information

You will be randomly divided into groups (exceptions are possible only for very good reasons).
The reason for this randomness is twofold:
(1) it will assure a broad distribution of skills and interest;
(2) it is a realistic scenario for later work in academia or industry.

Each group will be paired with another group, and they will act as review groups for each other.
Each pair of groups gets a tutor assigned and has 4 meetings, 45 minutes each, with this tutor during the course.
The tutor’s job is to give you general advice on anything you need as best as he or she can. The tutor’s job is not to formulate your project for you, nor to debug your code.
Should you experience problems with the group assembly, you need to take this up with your tutor as soon as possible.

To start your project you need a challenge you will address. You can get a challenge in one of the popular challenge websites, such a Kaggle, CodaLab, etc.

The first meeting with the tutor is to ensure that you’re embarking on a feasible project. Before this meeting you should:

Identify your challenge.
Write a two-paragraph proposal: one explaining the background of the project, possibly with references to relevant literature, and another outlining what you aim to do and why.
Discuss your project with your review group and revise your proposal accordingly.
Send the proposal to your tutor.
Prepare for the meeting; prepare questions, comments, simulations, results, or whatever is relevant for a most efficient outcome of the meeting.

The following meetings with the tutor are to ensure that you have progress and direction so that you are on course to completion. Before these meetings you should:

Have ready an implementation of your model.
Have a set of preliminary results from simulations.
Have ready the outline of a report.
Have met with your review group to discuss model, results and report outline.
Send the report outline to your tutor.
Prepare for the meeting; prepare questions, comments, simulations, results, or whatever is relevant for a most efficient outcome of the meeting.

Presentation

The presentation of the project should take maximum 8 minutes and be prerecorded (MOV format), with an additional 10 minutes for questions and discussion.
Put the emphasis on what your problem formulation is, a general discussion about how you tackled it, what problems you had, and what your conclusions are.

Report

The report should be written in RevTex 4.2 (see https://journals.aps.org/revtex), using the format of Physical Review X with a limit of 8 pages in double-column reprint format (including pictures and references). If needed, you can have an extra document containing an unlimited amount of Supplementary Materials.

Structure your report like a scientific article, with an abstract summarising the rationale and results of the project; an introduction shortly explaining its background and motivating why the question is interesting; methods and/or results section(s) describing your model and what you do with it; a discussion section; and ending with a conclusion. Put some work into the discussion and conclusion sections. This is where you demonstrate that you truly understand the implications of your work, including shortcomings and uncertainties. It is important that the discussion does not fall out as a simple summary of what the figures show.

Write in plain language and write enough to say what you need to say. Don't think "the report feels short, better throw in some extra figures." If you can say the same thing in fewer words, do so.

Be sure to reference any source you use. The report should be readable and understandable on its own, but there is no need to reproduce for example derivations of equations from your sources; a citation is enough.

Figures are an important part of the report, but only those that substantially contribute to your analysis should be included. Make sure that each figure is well designed with informative captions (not just "Fig. X shows how quantity A depends on quantity B"). If you find it hard to do this, you probably are not clear on why you include an figure, so cut it out.

Most importantly, use your own judgement and try to write a report you would like to read.

Evaluation criteria for report

Criteria that will be considered are:

Structure: Is the project report well structured (see above)?
Motivation: Does the project have a clear research question?
Originality: How original is the research question?
Background: Is the background of the research question well explained, including references to the literature?
Methods: Are the methods and analysis appropriate for convincingly answering the research question?
Execution: Does the project answer the initial or iterated research question?
Figures: Are the figures included informative with descriptive captions?
Format: Is the format of the report as required?
Outlook: Are the implications of the results clearly stated?
Contributions: Is there a contributions statement in the report?

Note that the research question does not have to be answered in the affirmative. A negative result is equally valid as long as an analysis is carried out that properly explains the negative result and what eventually would work better.

Evaluation criteria for presentation