Big Data Engineering

Dear students,

we have just published the results of the exam. You can view your results on your personal status page.

The exam inspection will take place next Tuesday, 06.08.2024, between 14:00 and 16:00, in E1 1 3.06.

In order to ensure that not all students show up at the same time, we have created time slots of 20 minutes. Please register for a time slot via your personal status page until next Monday, 05.06.2024, 23:59 if you wish to participate in the exam inspection. Remember to bring your student ID.

Kind regards,
Your BDE Team

Dear students,

the exam registration is closed. We have imported all registrations from LSF/HISPOS and FlexNow into our CMS. Thus, all registered students are now in the system. Please check on your personal status page if you are correctly registered for the exam. You are allowed to participate in the exam, if you are both admitted and registered. In this case, you should see the following on your personal status page.

If there are any discrepancies, please contact Luca Gretscher immediately.

Additionally, we have distributed all participating students to the different lecture halls. The exam will take place next Thursday, 01.08.2024, from 10:00 - 12:00 in the following lecture halls:

•     Building E2 2, Lecture Hall 0.01 (AudiMo, Günter-Hotz)
•     Building E2 5, Lecture Hall 01    (Mathe HS I)
•     Building E1 3, Lecture Hall 0.02 (HS002)

You can find your assigned lecture hall in the CMS on your personal status page under Exam Seats. Only the room is important here, not the exact seat. You are free to choose your seat.
 

General Information and Procedure

• The exam will start at 10:00am and will last 120 minutes.
• Be in front of your respective lecture hall at 9:45am at the latest.
• You may use two A4 sheets with hand-written notes (front and back) and a non-programmable calculator. All other materials are not permitted.
• Use a document-safe, permanent pen such as a ballpoint pen.
• We will distribute the exams to the individual seats before you enter the lecture hall. Only take a seat where there is an exam.
• You can read the instructions on the cover page directly, but do not open the exam booklet until your supervisor announces the start of the exam. Premature opening will be considered as attempted cheating.
• Place your bags on the sides of the lecture hall.
• Take your student ID card with you to your seat.

All further information about the exam procedure will be announced by the supervisor before the exam begins.

If anything is still unclear, please use our forum.

Kind regards,
Your BDE Team

Dear students,

as mentioned in todays lecture, we re-opened the evaluation for our course again since we received only very few feedback previously. However, as we try to improve our courses continuously, we would appreciate your feedback. You can find the links to the evaluation of the lecture and tutorials under Information in the CMS.

The evaluation will be available until tomorrow 26.07.2024.

Kind regards,
Your BDE Team

Dear students,

all points of the assignment sheets are entered in the CMS. On your personal status page you can see if you are admitted to the exams. Note that you need to pass the assignments and the project to be admitted.

You can still register for the exam as well as withdraw your registration until this Thursday, 25.07.2024. For more details regarding exam registration see this news.

If there are problems regarding the CMS registration, please contact Luca Gretscher.
In case of problems with the exam registration in general, please contact your responsible examination office.

General Notes and Tools Allowed for the Exam:

• The exam will be in English and you may give your answers in English as well as in German.
• You may only use blue or black document-proof pens, such as ballpoint pens.
• A non-programmable calculator and two handwritten DIN A4 sheets (front and back) are permitted.

If something is still unclear, please use our forum.

Kind regards,
Your BDE Team

Dear students,

for the last lecture on Thursday, July 25th, we will change the mode to a Q&A session:

    •    We will upload the summary slides and the summary lecture recording from last year.
    •    We will give a very short recap on the content presented throughout the semester.
    •    We will answer questions from you in the plenum. Therefore, we ask you to post questions in this forum post until Tuesday, July 23rd such that we can prepare a bit.
    •    Afterwards, our team will be there to answer more of your questions like in an office hour.

Best regards,
Your BDE Team

Dear students,

since we now entered the results of the project, the first students are officially admitted to the exam.

the exam will take place on 01.08.2024 from 10:00 - 12:00 in the lecture halls E2 2 - HS 0.01 (Günter-Hotz-Hörsaal), E1 3 - HS 001 + HS 002, and E2 5 HS I. We will announce which lecture hall you have been assigned to via the CMS on your personal status page after the registration deadline in the week before the exam.

The exam will start at 10:00am and will last 120 minutes. Therefore, please be in front of your respective lecture hall at 9:45am at the latest so that we can start on time.

The registration for the exam depends on your course of study. On your personal status page you can see whether you have to register via LSF/HISPOS or the CMS (or FlexNow which you hopefully already did as stated in this news). For LSF/HISPOS and CMS, you can register as well as withdraw your registration until one week before the exam (25.07.2024). This means, if you plan to participate in the exam, it is best to register right away and if necessary, withdraw your registration again in time.

Please note that there are individual registrations for the main exam and the re-exam.

Also note that the registration in CMS is only possible once you have successfully obtained the admission (50% of the assignments points, 50% of the project points, and no more than 2 assignment sheets with 0 points). If there are problems regarding the CMS registration, please contact Luca Gretscher.

In case of problems with the exam registration in general, please contact your responsible examination office.

Kind regards,
Your BDE Team

Dear students,

we have just published the timetable for the project code reviews this week under Materials -> IMDb (Part 3). Please be there with the whole team at least 10 minutes before your actual timeslot and wait in front of the indicated room!

In the code reviews, we will ask you multiple questions regarding the project, mainly about tasks T5-T9 but also about T1-T4. Keep in mind that every team member must know and understand all code. We will pick a random team member to answer a specific question. Team members who are not able to explain parts of the code do not get points for the project.

If all team members and the tutor present agree, code reviews may be held in German or in English.

You do not need to bring a laptop with your code, we will show your submission on our own laptops. Please bring your student ID for checking your identity!

Kind regards,
Your BDE Team

Dear all,

due to a medical emergency in my family I cannot to give the lecture on Thursday, June 27th.

We need to fall back to the recording from last year.

It can be found here:

https://www.youtube.com/watch?v=CU4sihiAl_M

Best regards,

Jens Dittrich

PS: If you haven't done so already, please do not forget to fill in the lecture evaluation (possible until end of June 27th). See materials section 07 in CMS.

Dear students,

due to an issue with task T2c (see forum posts), we have just uploaded a new version of the project under Materials -> IMDb (Part 3). Note that only the project description and the test file `famesocialnetwork/tests.py` differ slightly from the original version. So do not worry about merging your current progress into the updated version, just replace the test file! According to the updated project description, you do not need to change the MyLoginView class in any way (this makes the task actually easier), thus, we will also replace the corresponding file with a fresh version when determining your final test results.

Additionally, here are some clarifications:
    •    The bonus points of tasks T8 and T9 are only for the project, not for the final grade of the entire course.
    •    The code review is mandatory for all team members regardless of the number of points achieved through the unit/end-to-end tests.
    •    Only register your team for a single code review slot, otherwise not enough slots will be available for other teams. Keep in mind that only one team member registers for the entire team.

Kind regards,
Your BDE Team

Dear students,

we have just uploaded the project under Materials -> IMDb (Part 3). A detailed project description including instructions for the submission can be found in the project.pdf file in the root directory of the project. The submission deadline for the project is Sunday, 07.07.24, 23:59!

Additionally, each group must participate in a code review after the project submission. We offer eight code review slots:
    •    Slot 1: Wednesday 10.07.24, 10:00 - 12:00
    •    Slot 2: Wednesday 10.07.24, 12:00 - 14:00
    •    Slot 3: Thursday     11.07.24, 08:00 - 10:00
    •    Slot 4: Thursday     11.07.24, 12:00 - 14:00
    •    Slot 5: Thursday     11.07.24, 14:00 - 16:00
    •    Slot 6: Thursday     11.07.24, 16:00 - 18:00
    •    Slot 7: Friday          12.07.24, 08:00 - 10:00
    •    Slot 8: Friday          12.07.24, 10:00 - 12:00

More detailed information about the code reviews will be published at the end of the project, but keep in mind that every team members must know and understand all code. Team members who are not able to explain parts of the code do not get points for the project.

Please register for the code review on your personal status page by Sunday, 07.07.24, 23:59! Note that only one team member is allowed to register for the entire team, so please coordinate internally! Registration is on a first-come-first-served basis.

Kind regards,
Your BDE Team

Dear students,

due to the talk by Richard Hipp next Tuesday, we will adapt tutorials 5&6. If you want to attend the talk, you are welcome to attend another tutorial slot next week. However, we will still offer a single tutorial in the regular time slot in E1 1, 3.06 for those who do not want to attend the talk.

Kind regards,
Your BDE Team

Dear students,

we would like to remind you that the deadline for creating or joining a team for the project submission is tomorrow morning at 12am. If you are still looking for a team, you can use the forum.

If you correctly joined a team, only one team member has to upload your submission. Please make sure that every team member is able to see the uploaded submission on their personal status page. Only then, all team members will receive the points.

Please note, that only submissions in groups of 3 to 4 students are allowed. Submissions of smaller groups will not be graded.

Kind regards,
Your BDE Team

Title:
How SQLite Works

Speaker:
Richard Hipp

When:
Tuesday, June 25th, 12:15-14:00

Where:
GHH

Abstract:
If you measure success by the number of deployments, SQLite dominates the
database world. There are more instances of SQLite running today than all
other database engines combined.  Why is this?  How did SQLite come to be
so widely used?  What makes SQLite different?

This talk attempts to answer those questions by reviewing the unusual
design and implementation choices in SQLite, and showing how those
choices make SQLite well suited for mobile phones, appliances,
applications, and the "internet of things" where SQLite is often found.
The talk will delve into SQLite's use of bytecode, the techniques used
to achieve ACID power-safe transactions, and the query planning
algorithms.

This talk is targeted toward listeners who are familiar with
other database implementations, but it also accessible to people with
only a general computer programming background and who just want to
know more about how an SQL database engine works behind the scenes.

Short Bio:
Richard Hipp is an entrepreneur based in Charlotte, NC, USA, and the founder
and lead developer for SQLite as well as several other open-source projects.
He has been an invited speaker at universities around the world, including
MIT, Harvard, CMU, NYU, IIT Kharagpur, Politechnika Gdańska, and others.

Richard never intended to become a "database guy".  His PhD dissertation
(Duke University, 1992) was in computational linguistics, and his
undergraduate studies were in electrical engineering and digital signal
processing.  Because he did not come up inside the database community,
Richard brings a different perspective to database design, implementation,
and usage.

When he isn't sitting at the keyboard writing code (his usual habitat),
Richard can often be found out on long-distance runs or cooking vegetarian
meals with his wife, Ginger.

Host:
Jens Dittrich

Dear students,

the programming project will start in about two weeks. Analogously to the assignments, we allow submitting the project in teams of 3 to 4 students. Please note, that submissions of smaller groups will not be graded and thus the exam admission cannot be achieved. It is permitted to form teams different from those for the assignments, however, we recommend submitting the project in the same team. In any case, you have to additionally register your team!

To register your team, you have to use the Team Groupings functionality of the CMS. This means for you that a team member creates a group in the CMS on the personal status page and invites all other team members to the group. You can create your team until Thursday 20.06.24 at 12:00am, after that it is not possible to change the team.

If you haven't found a team yet, you can use our forum to find one.

Kind regards,
Your BDE Team

Dear students,

since the number of participants in the tutorials is continuously decreasing, we have decided to merge some of the tutorials that take place at the same time. Concretely, we will be merging the tutorials 1 & 2, the tutorials 3 & 4, and the tutorials 7 & 8 from now on. Here is an overview of the tutorial slots starting next week.

    •    Tutorial 1 & 2: Monday, 10:15 in E1 1, 3.06
    •    Tutorial 3 & 4: Monday, 12:15 in E1 1, 3.06
    •    Tutorial 5: Tuesday, 12:15 in E1 1, 3.06
    •    Tutorial 6: Tuesday, 12:15 in E1 1, 2.06
    •    Tutorial 7 & 8: Tuesday, 14:15 in E1 1, 3.06

You can find a concise overview in our Timetable as well.

Kind regards,
Your BDE Team

Dear students,

since the number of participants in the tutorials is lower than expected, we have decided to merge tutorial 9 with tutorials 5 & 6 that take place at the same time. Therefore, we reassigned every student of tutorial 9 to one of the other two tutorials. Please check on your Personal Status page where your new tutorial takes place.

Kind regards,
Your BDE Team

Dear students,

we recognized that a few students submitted in groups which did not meet the requirement of having at least 3 team members. Thus, these submission unfortunately will not be graded. There were also some single member teams that did not submit.

However, as we want to give you a fair chance for submitting the next assignment, we would like to offer the following possibilities to those of you that are still looking for a team. Please choose one of the two options.

• You can search a team by yourself. There are still open threads in the forum for that. Write an email to Luca Gretscher until Monday, 06.05.2024, 10am and state the matriculation numbers of all your team members. Please also state whether the team was already created and some students want to join or whether a new team should be created.
• If you do not find a team but want to submit the assignments anyway, please also write an email to Luca Gretscher until Monday, 06.05.2024, 10am and state your matriculation number. We will then create new teams for those students and randomly assign you to one of them.

Please note, that still only submissions in groups of 3 to 4 students are allowed. Submissions of smaller groups will not be graded.

Kind regards,
Your BDE Team

Dear students,

we would like to remind you that the deadline for creating or joining a team for the assignment submissions is tomorrow morning at 10am. If you are still looking for a team, you can use the forum. There are still open threads.

If you correctly joined a team, only one team member has to upload your submission. Please make sure that every team member is able to see the uploaded submission on their personal status page. Only then, all team members will receive the points.

Please note, that only submissions in groups of 3 to 4 students are allowed. Submissions of smaller groups will not be graded.

Kind regards,
Your BDE Team

Dear students,

we have now assigned you to your tutorial groups. You can find your assigned tutorial on your personal status page. The tutorials will start next week. More information can be found in the Timetable.

If you were assigned to Tutorial 9 (Monday, 00:00 No Tutorial) but still want to attend a tutorial at some point, you are welcome to attend the tutorial on Tuesdays 12:15 - 13:45 in the Günter-Hotz lecture hall.

Please bring your laptop to the tutorials since we will do some coding exercises (at least next week). Additionally, we will upload the tutorial slides before each tutorial.

Kind regards,
Your BDE Team

Dear students,

we have just uploaded Assignment 01. Instructions for the submission can be found at the end of the document.

Please note that the submission is only allowed in teams of 3 to 4 students. Furthermore, we ask that you try to form teams of exactly 4 students to reduce the number of submissions.

To register your team, you have to use the Team Groupings functionality of the CMS. This means for you that a team member creates a group in the CMS on the personal status page and invites all other team members to the group. You can create your team until Thursday 02.05.24 at 10:00am, after that it is not possible to change the team. If you need to change your team during the semester, please contact your tutor.

If you haven't found a team yet, you can use our forum to find one.

Kind regards,
Your BDE Team

Dear students,

we would like to remind you that you can change your preferences for the tutorials only until tonight.

We offer nine regular tutorials at the following times:
    •    Slot 1: Monday, 00:00 No Tutorial: This is a dummy tutorial! Please give preference to this tutorial if you do not plan to attend the tutorials in general and rule out the other tutorial choices.
    •    Slot 2: Monday, 10:15 Tutorial: Two regular tutorials at this time slot in E1.1 R2.06 and E1.1 R3.06
    •    Slot 3: Monday, 12:15 Tutorial: Two regular tutorials at this time slot in E1.1 R2.06 and E1.1 R3.06
    •    Slot 4: Tuesday, 12:15 Tutorial: Three regular tutorials at this time slot in E1.1 R2.06, E1.1 R3.06, and E2.2 0.01 (GHH)
    •    Slot 5: Tuesday, 14:15 Tutorial: Two regular tutorials at this time slot in E1.1 R2.06 and E1.1 R3.06

In the tutorials, we will in general briefly discuss the solution of the previous assignment sheet. However, the main focus will be a recap of the lecture material by providing additional exercises.
Please note, that both the solutions of the assignment sheets as well as the tutorial slides including solutions will be timely uploaded to our Materials.

Because of this, experience has shown that some students will not attend the tutorials in person. Therefore, we also created an additional (pseudo) tutorial (Slot 1: Monday, 00:00). Please give preference to this tutorial if you do not plan to attend the tutorials in general and rule out the other tutorial choices. With this, we can allow students that plan to attend, to more likely get their preferred choice. Furthermore, we can better assess how many in-person tutorials we really need. Of course, you can always attend a tutorial if you change your mind.

Please submit your tutorial preferences on your personal status page by 23:59 today!

Kind regards,
Your BDE Team

Dear students,

in the course of this lecture, we will use Python and in particular Jupyter Notebooks to explain various concepts. For a few notebooks, we also require additional software (e.g. PostgreSQL). Therefore, we provide a Virtual Machine using Vagrant that has all the required software and systems already installed. You can find detailed instructions at the following Github repository. The link is also available under Information > Vagrant. We also provide a small FAQ.

As there are problems with Vagrant on Apple Silicon chips, we also created a Docker image which can be used instead of the Vagrant VM. You can find detailed instructions at the following Github repository. The link is also available under Information > Docker.

We spent a lot of time preparing the tools and the instructions. Please read them carefully before asking questions.

Please note, that you are also able to run most of the notebooks by just having a Python installation with the required packages.

If you still have problems with the installation, you can always ask a question in the forum (bde24 > Technical Problems).

Kind regards,
Your BDE Team