Big Data Engineering

Dear students,

we have just published the timetable for the project code reviews this week under Materials -> IMDb (Part 3). Please be there with the whole team at least 10 minutes before your actual timeslot and wait in front of the indicated room!

In the code reviews, we will ask you multiple questions regarding the project, mainly about tasks T5-T9 but also about T1-T4. Keep in mind that every team member must know and understand all code. We will pick a random team member to answer a specific question. Team members who are not able to explain parts of the code do not get points for the project.

If all team members and the tutor present agree, code reviews may be held in German or in English.

You do not need to bring a laptop with your code, we will show your submission on our own laptops. Please bring your student ID for checking your identity!

Kind regards,
Your BDE Team

Dear all,

due to a medical emergency in my family I cannot to give the lecture on Thursday, June 27th.

We need to fall back to the recording from last year.

It can be found here:

https://www.youtube.com/watch?v=CU4sihiAl_M

Best regards,

Jens Dittrich

 

PS: If you haven't done so already, please do not forget to fill in the lecture evaluation (possible until end of June 27th). See materials section 07 in CMS.

Dear students,

due to an issue with task T2c (see forum posts), we have just uploaded a new version of the project under Materials -> IMDb (Part 3). Note that only the project description and the test file `famesocialnetwork/tests.py` differ slightly from the original version. So do not worry about merging your current progress into the updated version, just replace the test file! According to the updated project description, you do not need to change the MyLoginView class in any way (this makes the task actually easier), thus, we will also replace the corresponding file with a fresh version when determining your final test results.

Additionally, here are some clarifications:
    •    The bonus points of tasks T8 and T9 are only for the project, not for the final grade of the entire course.
    •    The code review is mandatory for all team members regardless of the number of points achieved through the unit/end-to-end tests.
    •    Only register your team for a single code review slot, otherwise not enough slots will be available for other teams. Keep in mind that only one team member registers for the entire team.

Kind regards,
Your BDE Team

Dear students,

we have just uploaded the project under Materials -> IMDb (Part 3). A detailed project description including instructions for the submission can be found in the project.pdf file in the root directory of the project. The submission deadline for the project is Sunday, 07.07.24, 23:59!

Additionally, each group must participate in a code review after the project submission. We offer eight code review slots:
    •    Slot 1: Wednesday 10.07.24, 10:00 - 12:00
    •    Slot 2: Wednesday 10.07.24, 12:00 - 14:00
    •    Slot 3: Thursday     11.07.24, 08:00 - 10:00
    •    Slot 4: Thursday     11.07.24, 12:00 - 14:00
    •    Slot 5: Thursday     11.07.24, 14:00 - 16:00
    •    Slot 6: Thursday     11.07.24, 16:00 - 18:00
    •    Slot 7: Friday          12.07.24, 08:00 - 10:00
    •    Slot 8: Friday          12.07.24, 10:00 - 12:00

More detailed information about the code reviews will be published at the end of the project, but keep in mind that every team members must know and understand all code. Team members who are not able to explain parts of the code do not get points for the project.

Please register for the code review on your personal status page by Sunday, 07.07.24, 23:59! Note that only one team member is allowed to register for the entire team, so please coordinate internally! Registration is on a first-come-first-served basis.

Kind regards,
Your BDE Team

Dear students,

due to the talk by Richard Hipp next Tuesday, we will adapt tutorials 5&6. If you want to attend the talk, you are welcome to attend another tutorial slot next week. However, we will still offer a single tutorial in the regular time slot in E1 1, 3.06 for those who do not want to attend the talk.

Kind regards,
Your BDE Team

Dear students,

we would like to remind you that the deadline for creating or joining a team for the project submission is tomorrow morning at 12am. If you are still looking for a team, you can use the forum.

If you correctly joined a team, only one team member has to upload your submission. Please make sure that every team member is able to see the uploaded submission on their personal status page. Only then, all team members will receive the points.

Please note, that only submissions in groups of 3 to 4 students are allowed. Submissions of smaller groups will not be graded.

Kind regards,
Your BDE Team

Title:
How SQLite Works

Speaker:
Richard Hipp

When:
Tuesday, June 25th, 12:15-14:00

Where:
GHH

Abstract:
If you measure success by the number of deployments, SQLite dominates the
database world. There are more instances of SQLite running today than all
other database engines combined.  Why is this?  How did SQLite come to be
so widely used?  What makes SQLite different?

This talk attempts to answer those questions by reviewing the unusual
design and implementation choices in SQLite, and showing how those
choices make SQLite well suited for mobile phones, appliances,
applications, and the "internet of things" where SQLite is often found.
The talk will delve into SQLite's use of bytecode, the techniques used
to achieve ACID power-safe transactions, and the query planning
algorithms.

This talk is targeted toward listeners who are familiar with
other database implementations, but it also accessible to people with
only a general computer programming background and who just want to
know more about how an SQL database engine works behind the scenes.

Short Bio:
Richard Hipp is an entrepreneur based in Charlotte, NC, USA, and the founder
and lead developer for SQLite as well as several other open-source projects.
He has been an invited speaker at universities around the world, including
MIT, Harvard, CMU, NYU, IIT Kharagpur, Politechnika Gdańska, and others.

Richard never intended to become a "database guy".  His PhD dissertation
(Duke University, 1992) was in computational linguistics, and his
undergraduate studies were in electrical engineering and digital signal
processing.  Because he did not come up inside the database community,
Richard brings a different perspective to database design, implementation,
and usage.

When he isn't sitting at the keyboard writing code (his usual habitat),
Richard can often be found out on long-distance runs or cooking vegetarian
meals with his wife, Ginger.

Host:
Jens Dittrich

 

Dear students,

the programming project will start in about two weeks. Analogously to the assignments, we allow submitting the project in teams of 3 to 4 students. Please note, that submissions of smaller groups will not be graded and thus the exam admission cannot be achieved. It is permitted to form teams different from those for the assignments, however, we recommend submitting the project in the same team. In any case, you have to additionally register your team!

To register your team, you have to use the Team Groupings functionality of the CMS. This means for you that a team member creates a group in the CMS on the personal status page and invites all other team members to the group. You can create your team until Thursday 20.06.24 at 12:00am, after that it is not possible to change the team.

If you haven't found a team yet, you can use our forum to find one.

Kind regards,
Your BDE Team

Dear students,

since the number of participants in the tutorials is continuously decreasing, we have decided to merge some of the tutorials that take place at the same time. Concretely, we will be merging the tutorials 1 & 2, the tutorials 3 & 4, and the tutorials 7 & 8 from now on. Here is an overview of the tutorial slots starting next week.

    •    Tutorial 1 & 2: Monday, 10:15 in E1 1, 3.06
    •    Tutorial 3 & 4: Monday, 12:15 in E1 1, 3.06
    •    Tutorial 5: Tuesday, 12:15 in E1 1, 3.06
    •    Tutorial 6: Tuesday, 12:15 in E1 1, 2.06
    •    Tutorial 7 & 8: Tuesday, 14:15 in E1 1, 3.06

You can find a concise overview in our Timetable as well.

Kind regards,
Your BDE Team

Dear students,

since the number of participants in the tutorials is lower than expected, we have decided to merge tutorial 9 with tutorials 5 & 6 that take place at the same time. Therefore, we reassigned every student of tutorial 9 to one of the other two tutorials. Please check on your Personal Status page where your new tutorial takes place.

Kind regards,
Your BDE Team

Dear students,

we recognized that a few students submitted in groups which did not meet the requirement of having at least 3 team members. Thus, these submission unfortunately will not be graded. There were also some single member teams that did not submit.

However, as we want to give you a fair chance for submitting the next assignment, we would like to offer the following possibilities to those of you that are still looking for a team. Please choose one of the two options.

• You can search a team by yourself. There are still open threads in the forum for that. Write an email to Luca Gretscher until Monday, 06.05.2024, 10am and state the matriculation numbers of all your team members. Please also state whether the team was already created and some students want to join or whether a new team should be created.
• If you do not find a team but want to submit the assignments anyway, please also write an email to Luca Gretscher until Monday, 06.05.2024, 10am and state your matriculation number. We will then create new teams for those students and randomly assign you to one of them.

Please note, that still only submissions in groups of 3 to 4 students are allowed. Submissions of smaller groups will not be graded.

Kind regards,
Your BDE Team

Dear students,

we would like to remind you that the deadline for creating or joining a team for the assignment submissions is tomorrow morning at 10am. If you are still looking for a team, you can use the forum. There are still open threads.

If you correctly joined a team, only one team member has to upload your submission. Please make sure that every team member is able to see the uploaded submission on their personal status page. Only then, all team members will receive the points.

Please note, that only submissions in groups of 3 to 4 students are allowed. Submissions of smaller groups will not be graded.

Kind regards,
Your BDE Team

Dear students,

we have now assigned you to your tutorial groups. You can find your assigned tutorial on your personal status page. The tutorials will start next week. More information can be found in the Timetable.

If you were assigned to Tutorial 9 (Monday, 00:00 No Tutorial) but still want to attend a tutorial at some point, you are welcome to attend the tutorial on Tuesdays 12:15 - 13:45 in the Günter-Hotz lecture hall.

Please bring your laptop to the tutorials since we will do some coding exercises (at least next week). Additionally, we will upload the tutorial slides before each tutorial.

Kind regards,
Your BDE Team

Dear students,

we have just uploaded Assignment 01. Instructions for the submission can be found at the end of the document.

Please note that the submission is only allowed in teams of 3 to 4 students. Furthermore, we ask that you try to form teams of exactly 4 students to reduce the number of submissions.

To register your team, you have to use the Team Groupings functionality of the CMS. This means for you that a team member creates a group in the CMS on the personal status page and invites all other team members to the group. You can create your team until Thursday 02.05.24 at 10:00am, after that it is not possible to change the team. If you need to change your team during the semester, please contact your tutor.

If you haven't found a team yet, you can use our forum to find one.

Kind regards,
Your BDE Team

Dear students,

we would like to remind you that you can change your preferences for the tutorials only until tonight.

We offer nine regular tutorials at the following times:
    •    Slot 1: Monday, 00:00 No Tutorial: This is a dummy tutorial! Please give preference to this tutorial if you do not plan to attend the tutorials in general and rule out the other tutorial choices.
    •    Slot 2: Monday, 10:15 Tutorial: Two regular tutorials at this time slot in E1.1 R2.06 and E1.1 R3.06
    •    Slot 3: Monday, 12:15 Tutorial: Two regular tutorials at this time slot in E1.1 R2.06 and E1.1 R3.06
    •    Slot 4: Tuesday, 12:15 Tutorial: Three regular tutorials at this time slot in E1.1 R2.06, E1.1 R3.06, and E2.2 0.01 (GHH)
    •    Slot 5: Tuesday, 14:15 Tutorial: Two regular tutorials at this time slot in E1.1 R2.06 and E1.1 R3.06

In the tutorials, we will in general briefly discuss the solution of the previous assignment sheet. However, the main focus will be a recap of the lecture material by providing additional exercises.
Please note, that both the solutions of the assignment sheets as well as the tutorial slides including solutions will be timely uploaded to our Materials.

Because of this, experience has shown that some students will not attend the tutorials in person. Therefore, we also created an additional (pseudo) tutorial (Slot 1: Monday, 00:00). Please give preference to this tutorial if you do not plan to attend the tutorials in general and rule out the other tutorial choices. With this, we can allow students that plan to attend, to more likely get their preferred choice. Furthermore, we can better assess how many in-person tutorials we really need. Of course, you can always attend a tutorial if you change your mind.

Please submit your tutorial preferences on your personal status page by 23:59 today!

Kind regards,
Your BDE Team

Dear students,

in the course of this lecture, we will use Python and in particular Jupyter Notebooks to explain various concepts. For a few notebooks, we also require additional software (e.g. PostgreSQL). Therefore, we provide a Virtual Machine using Vagrant that has all the required software and systems already installed. You can find detailed instructions at the following Github repository. The link is also available under Information > Vagrant. We also provide a small FAQ.

As there are problems with Vagrant on Apple Silicon chips, we also created a Docker image which can be used instead of the Vagrant VM. You can find detailed instructions at the following Github repository. The link is also available under Information > Docker.

We spent a lot of time preparing the tools and the instructions. Please read them carefully before asking questions.

Please note, that you are also able to run most of the notebooks by just having a Python installation with the required packages.

If you still have problems with the installation, you can always ask a question in the forum (bde24 > Technical Problems).

Kind regards,
Your BDE Team