MDK: Modern Database Concepts
Basic Information
Homework Deadlines
- 00 - Topic selection: Saturday 4. 4. 2020 until 23:59
- 01 - XPath: Saturday 18. 4. 2020 until 23:59
- 02 - XQuery: Sunday 19. 4. 2020 until 23:59
- 03 - SPARQL: Saturday 16. 5. 2020 until 23:59
- 04 - MapReduce: Sunday 17. 5. 2020 until 23:59
- 05 - Riak: Saturday 30. 5. 2020 until 23:59
- 06 - Cassandra: Sunday 31. 5. 2020 until 23:59
- 07 - MongoDB: Saturday 20. 6. 2020 until 23:59
- 08 - Neo4j: Sunday 21. 6. 2020 until 23:59
Classes and Materials
Onsite online classes Friday 3. 4. 2020
- 13:30 - 15:00: Lecture 01 - Introduction: Big Data, NoSQL Databases
- 15:15 - 16:45: Lecture 02 - Data Formats: XML, JSON, BSON, RDF
- 17:00 - 18:30: Practical Class 01 - Formats
Onsite online classes Saturday 4. 4. 2020
- 08:15 - 09:45: Lecture 03 - XML Databases: XPath
- 10:00 - 11:30: Practical Class 02 - XPath
- 13:30 - 15:00: Lecture 04 - XML Databases: XQuery
- 15:15 - 16:45: Practical Class 03 - XQuery
- 17:00 - 18:30: Lecture 05 - Basic Principles: Scaling, Sharding, Replication, CAP Theorem, Consistency
Online consultations
- Friday 17. 4. 2020: 13:30 - 15:00
Onsite online classes Friday 24. 4. 2020
- 13:30 - 15:00: Lecture 06 - RDF Stores: SPARQL
- 15:15 - 16:45: Practical Class 04 - SPARQL
- 17:00 - 18:30: Lecture 07 - Apache Hadoop: MapReduce, HDFS
Onsite online classes Saturday 25. 4. 2020
Online consultations
- Friday 15. 5. 2020: 13:30 - 15:00
- Friday 29. 5. 2020: 13:30 - 15:00
Onsite online classes Friday 5. 6. 2020
Onsite online classes Saturday 6. 6. 2020
- 08:15 - 09:45: Practical Class 10 - MongoDB
- 10:00 - 11:30: Lecture 11 - Graph Databases: Neo4j: Traversal Framework
- 13:30 - 15:00: Lecture 12 - Graph Databases: Neo4j: Cypher
- 15:15 - 16:45: Practical Class 11 - Neo4j
- 17:00 - 18:30: Practical Class 12 - Neo4j
Online consultations
- Friday 19. 6. 2020: 13:30 - 15:00
Class Recordings
- Recordings of online classes are accessible via a user name student and the same password as for the online meetings
- Recordings can only be used for personal study purposes and may not be published or otherwise disseminated in any way
Lectures
Practical Classes
- 03. 04. 2020: 01 - Formats (43 MB)
- 04. 04. 2020: 02 - XPath (87 MB)
- 04. 04. 2020: 03 - XQuery (37 MB)
- 24. 04. 2020: 04 - SPARQL (76 MB)
- 25. 04. 2020: 05 - MapReduce (75 MB)
- 25. 04. 2020: 06 - RiakKV (66 MB)
- 25. 04. 2020: 07 - Cassandra (62 MB)
- 05. 06. 2020: 08 - Redis (80 MB)
- 05. 06. 2020: 09 - MongoDB (54 MB)
- 06. 06. 2020: 10 - MongoDB (65 MB)
- 06. 06. 2020: 11 - Neo4j (58 MB)
- 06. 06. 2020: 12 - Neo4j (58 MB)
Formal Requirements
- Attendance during lectures as well as practical classes is not compulsory, yet recommended
- Altogether 8 individual voluntary homework assignments will take place during the semester
- Everyone needs to choose their distinct topic, not later than before the submission of the first assignment
- Possible topics could be library, cinema, cookbook, university, flights, etc.
- See the list below for additional suitable topics, feel free to choose your own topic
- Your homework solutions must be within the topic, original, realistic, and non-trivial
- Solutions can only be submitted via a script executed on the corresponding server
- Solutions are awarded by up to 20 or 15 points respectively, depending on the assignment
- In case of any shortcomings, fewer points will be awarded appropriately
- Solutions can be submitted even repeatedly, only the latest version is assessed
- Once a given assignment is assessed by the lecturer, it cannot be resubmitted once again
- Delays are penalized, one whole day by 5 points, shorter delays proportionally
- Should the delay be even longer, the penalty stays the same and does not further increase
- None of the homework assignments is compulsory, yet you are encouraged to submit all of them
Homework Assignments
- Preliminaries:
- NoSQL server: nosql.ms.mff.cuni.cz:42222
- Login and password: sent by e-mail at the beginning of the semester
- Tools:
- Submissions:
- Use sftp or WinSCP tools to upload your submission files to the NoSQL server
- Put these files into a directory ~/assignments/name/, where name is a name of a given homework
- I.e. xpath, xquery, sparql, mapreduce, riak, cassandra, mongodb, neo4j (case sensitive)
- Use ssh or PuTTY tools to open a remote shell connection to the NoSQL server
- Based on the instructions provided for a given homework assignment, verify that everything is working as expected
- Go to the ~/assignments/ directory and execute submit name, where name is once again the name of the homework
- Wait for the confirmation of success, otherwise your homework is not considered to be submitted
- Should any complications appear, send your solution by e-mail to martin.svoboda@extern.oth-regensburg.de
- Just for your convenience, you can check the submitted files in the ~/submissions/ directory
- Requirements:
- Respect the prescribed names of individual files to be submitted (case sensitive)
- Place all the files in the root directory of your submission
- Do not include shared libraries or any other files that were not explicitly requested
- Do not redirect or suppress both standard and error outputs in your shell scripts
- All your files must be syntactically correct and executable without errors
1: XPath
- Points: 20
- Assignment:
- Create an XML document with sample data from the domain of your individual topic
- Work with mutually interlinked entities of at least 3 different types (e.g. lines, flights and tickets)
- Insert data about at least 15 particular entities (e.g. 3 lines, 4 flights, 8 tickets)
- Create expressions for exactly 5 different XPath queries (i.e. not more, not less)
- Use each of the following constructs at least once
- Axes: descendant or descendant-or-self or // abbreviation
- Axes: ancestor(-or-self) or preceding(-sibling) or following(-sibling)
- Predicates (all of the following): path expression (existence test), position testing, value comparison, general comparison
- Requirements:
- Both the XML document and queries must be well-formed (i.e. syntactically correct)
- Put each XPath expression into a standalone file (e.g. xpath1.xp)
- Always add a comment describing the intended query meaning in natural language via (: comment :)
- Each query expression must be evaluated to a non-empty sequence
- Submission:
- data.xml: XML document with your data to be queried
- xpath1.xp, ..., xpath5.xp: files with XPath expressions
- Execution:
- Execute the following shell command to evaluate each individual XPath query expression
- saxonb-xquery -s $DataFile $QueryFile
- $DataFile is the input XML document to be queried, i.e. data.xml
- $QueryFile is a file with query expression to be evaluated, e.g. xpath1.xp
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Saturday 18. 4. 2020 until 23:59
2: XQuery
- Points: 20
- Assignment:
- Create an XML document with sample data from the domain of your individual topic
- Work with mutually interlinked entities of at least 3 different types (e.g. lines, flights and tickets)
- Insert data about at least 15 particular entities (e.g. 3 lines, 4 flights, 8 tickets)
- This document may (or may not) be identical to the one from the previous assignment on XPath
- Create expressions for exactly 5 different XQuery queries (that cannot be expressed solely using XPath)
- Use each of the following constructs at least once
- Direct or computed constructor
- FLWOR expression (with at least one for, let, where and order by clauses)
- Aggregate function (min, max, avg or sum)
- Conditional expression
- Existential or universal quantifier
- Requirements:
- Both the XML document and queries must be well-formed (i.e. syntactically correct)
- Put each XQuery expression into a standalone file (e.g. xquery1.xq)
- Always add a comment describing the intended query meaning in natural language via (: comment :)
- Each query expression must be evaluated to a non-empty sequence
- Submission:
- data.xml: XML document with your data to be queried
- xquery1.xq, ..., xquery5.xq: files with XQuery expressions
- Execution:
- Execute the following shell command to evaluate each individual XQuery query expression
- saxonb-xquery -s $DataFile $QueryFile
- $DataFile is the input XML document to be queried, i.e. data.xml
- $QueryFile is a file with query expression to be evaluated, e.g. xquery1.xq
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Sunday 19. 4. 2020 until 23:59
3: SPARQL
- Points: 20
- Assignment:
- Create a TTL document with sample RDF triples within your individual topic
- Use the Turtle notation in particular
- Work with mutually interlinked resources of at least 3 different types (e.g. lines, flights and tickets)
- Insert data about at least 15 particular resources (e.g. 3 lines, 4 flights, 8 tickets)
- Use each of the following constructs at least once
- Object list or predicate-object list
- Blank nodes (either using _ prefix or brackets [])
- Create expressions for exactly 5 different SPARQL queries (SELECT query form in particular)
- Use each of the following constructs at least once
- Basic graph pattern
- Group graph pattern
- Optional graph pattern (OPTIONAL)
- Alternative graph pattern (UNION)
- Difference graph pattern (MINUS)
- FILTER constraint
- Aggregation (GROUP BY with or without HAVING clause)
- Sorting (ORDER BY clause)
- Requirements:
- Both TTL document and queries must be well-formed (i.e. syntactically correct)
- Put each SPARQL query expression into a standalone file (e.g. query1.sparql)
- Always add a comment describing the intended query meaning in natural language via # comment
- Each query expression must be evaluated to a non-empty solution sequence
- Both the data file a query files must contain declarations of all prefixes used, including rdf: and similar
- Use @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . in your data file
- Use PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> in your query file
- Do not use FROM clauses in your queries, the input data file will automatically be accessible as the default graph
- Submission:
- data.ttl: TTL document with your RDF data to be queried
- query1.sparql, ..., query5.sparql: files with SPARQL query expressions
- Execution:
- Execute the following shell command to evaluate each individual SPARQL query expression
- sparql --data $DataFile --query $QueryFile
- $DataFile is the input RDF document to be queried, i.e. data.ttl
- $QueryFile is a file with query expression to be evaluated, e.g. query1.sparql
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Saturday 16. 5. 2020 until 23:59
4: MapReduce
- Points: 20
- Assignment:
- Create an input text file with sample data from the domain of your individual topic
- Insert realistic and non-trivial data about at least 10 entities of one type
- Put each of these entities on a separate line, i.e. assume that each line of the input file yields one input record
- Organize the actual entity attributes in whatever way you are able to easily parse
- E.g. Medvídek 2007 53 100 Trojan Macháček Vilhelmová corresponding to a pattern Movie Year Rating Length Actors...
- Implement a non-trivial MapReduce job
- Choose from aggregation, grouping, filtering or any other general MapReduce usage pattern
- Use WordCount.java source file as a basis for your own implementation
- Both the Map and Reduce functions should be non-trivial, each about 10 lines of code
- It is not necessary to implement the Combine function
- Comment the source file and also provide a description of the problem you are solving
- You may also create a shell script that allows for the execution of your entire MapReduce job
- I.e. compile source files, deploy input file, execute the actual job, retrieve its result, ...
- However, this script is not supposed to be submitted and serves just for your own convenience
- Even if you do so, it will not be used for the purpose of homework assessment in any way
- Requirements:
- You may split your MapReduce job implementation into multiple Java source files
- They all must be located in the submission root directory
- At least MapReduce.java source file with its public MapReduce class is required
- This class is expected to represent the main class of the entire MapReduce job
- Do not change the way how command line arguments are processed
- I.e. the only two arguments represent the input and output HDFS locations respectively
- Do not use packages in order to organize your Java source files
- Assume that only hadoop-common-3.1.1.jar and hadoop-mapreduce-client-core-3.1.1.jar libraries will be linked with your project
- Do not submit your Netbeans (or any other) project directory, do not submit Hadoop (or any other) libraries
- Use Java Standard Edition version 7 or newer
- You are free to use your /user/r192_login/ HDFS home directory for debugging
- Homework assessment will take place in a different dedicated HDFS directory
- Submission:
- readme.txt: description of the input data structure and objective of the MapReduce job
- input.txt: text file with your sample input data (i.e. only one input file is permitted)
- MapReduce.java and possibly additional *.java: Java source files with your MapReduce implementation
- output.txt: expected output of your MapReduce job (i.e. submit the result of the execution you performed by yourself)
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Sunday 17. 5. 2020 until 23:59
5: Riak
- Points: 15
- Assignment:
- Create a shell script that works with Riak database via its HTTP interface using the cURL tool
- Insert about 5 key-value objects of different entity types into each of 3 buckets
- You can work with any data format you like (e.g. text, JSON)
- However, always include content headers
- Include at least 5 meaningful links of at least 2 different tags
- Use a separate Link header for each individual link you want to define
- I.e., do not incorporate multiple links into a single Link header
- Execute 1 read and 1 update request
- Perform 2 link walking queries
- One with at least 1 navigational step
- Another with at least 2 navigational steps
- Remove all your objects at the end of your script (i.e. empty all your buckets)
- Requirements:
- Assume the following convention for names of your buckets: r192_login_bucket
- Replace login with your actual login name and bucket with the intended bucket name
- E.g. r192_svobom25_movies for a bucket of movies
- However, use the following convention when referencing your buckets in URLs: $1_bucket
- $1 is a script argument provided from outside of your script (see below)
- Its value will correspond to a prefix r192_login with login as your actual login name
- I.e., do not access your buckets directly so that this prefix can easily be changed
- Note that variables enclosed in single quotes will not be replaced by their values, use double quotes instead
- E.g., your link definition can than look like this: "Link: </buckets/$1_bucket/keys/key>; riaktag=\"type\""
- Explain the real-world meaning of your link walking queries (# comment)
- Make sure your shell script is executable at all (i.e. has the X permission assigned)
- If you are using WinSCP, just locate your file, go to the properties context menu and add the X permission for the owner
- If you are using PuTTY, execute the following command chmod u+x script.sh
- Also make sure your script can be executed repeatedly without failures
- Submission:
- script.sh: Bash script allowing to execute all the HTTP requests
- Execution:
- Execute the following shell command to evaluate the whole RIAK script
- ./script.sh $BucketPrefix
- $BucketPrefix is a prefix for names of your buckets, i.e. r192_login
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Saturday 30. 5. 2020 until 23:59
6: Cassandra
- Points: 15
- Assignment:
- Create a script (ordinary text file) with a sequence of CQL statements working with Cassandra database
- Define a schema for 2 tables for entities of different types
- Define at least one column for each of the following data types: tuple, list, set and map
- Insert about 5 rows into each of your tables
- Express at least 3 update statements
- You must perform replace, add and remove primitive operations (all of them) on columns of all collection types (all of them)
- I.e. you must involve at least altogether 9 different primitive operations on such columns
- Express 3 select statements
- Use WHERE and ORDER BY clauses at least once (both of them)
- Use ALLOW FILTERING in a query that cannot be evaluated without this instruction
- Create at least 1 secondary index
- Requirements:
- Only use your own keyspace when working on the assignment
- Name of this keyspace must be identical to your login name (r192_login)
- Do not create this keyspace in your script (assume it already exists)
- Do not switch to your keyspace when you are inside your script
- I.e. do not execute a USE command to change the active keyspace from within the script
- Specify the intended keyspace outside your script using command line options (see below)
- Note that a different dedicated keyspace will be used when assessing your homework
- You can assume that this keyspace will be completely empty at the beginning
- Submission:
- script.cql: text file with CQL statements
- Execution:
- Execute the following shell command to evaluate the whole CQL script
- cqlsh -k $KeyspaceName -f $ScriptFile
- $KeyspaceName is a name of keyspace that should be used (must already exist), e.g. r192_login
- $ScriptFile is a file with CQL queries to be executed, i.e. script.cql
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Sunday 31. 5. 2020 until 23:59
7: MongoDB
- Points: 20
- Assignment:
- Create a JavaScript script with a sequence of commands working with MongoDB database
- Explicitly create 2 collections for entities of different types
- I.e., create them using createCollection method
- Insert about 5 documents into each one of them
- These documents must be realistic, non-trivial, and with both embedded objects and arrays
- Interlink the documents using references
- Use both insert and save operations, each at least once
- Express 3 update operations (do not use save operation for this purpose)
- One without update operators
- One with at least 2 different update operators
- One based on the upsert mode
- Express 5 find queries (with non-trivial selections)
- Use at least one logical operator ($and, $or, $not)
- Use $elemMatch operator on array fields at least once
- Use both positive and negative projection (each at least once)
- Use sort modifier
- Describe the real-world meaning of all your queries in comments
- Express 1 MapReduce query (non-trivial, i.e. not easily expressed using ordinary find operation)
- Describe its meaning, contents of intermediate key-value pairs and the final output
- Note that reduce function must be associative, commutative, and idempotent
- Requirements:
- Call export LC_ALL=C in case you have difficulties in launching the mongo shell
- Only use your own database when working on the assignment
- Name of this database must be identical to your login name (r192_login)
- Do not switch to your database when you are inside your script
- I.e. do not execute USE database and nor db.getSiblingDB('database') commands
- Specify the intended database outside your script using command line options (see below)
- Note that a different dedicated database will be used when assessing your homework
- You can assume that this database will be completely empty at the beginning
- Print the output of your queries (find operations)
- Use db.collection.find().forEach(printjson); approach for this purpose
- Print the output of your MapReduce job using out: { inline: 1 } option
- I.e. do not redirect the output into a standalone collection
- Submission:
- script.js: JavaScript script with MongoDB database commands
- Execution:
- Execute the following shell command to evaluate the whole MongoDB script
- mongo $DatabaseName $ScriptFile
- $DatabaseName is a name of database that should be used, e.g. r192_login
- $ScriptFile is a file with MongoDB queries to be executed, i.e. script.js
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Saturday 20. 6. 2020 until 23:59
8: Neo4j
- Points: 20
- Assignment:
- Insert realistic nodes and relationships into your embedded Neo4j database
- Use a single CREATE statement for this purpose
- Insert altogether at least 10 nodes for entities of at least 2 different types (i.e. different labels)
- Insert altogether at least 15 relationships of at least 2 different types
- Include properties (both for nodes and relationships)
- Associate all your nodes with user-defined identifiers
- Express 5 Cypher query expressions
- Use at least once MATCH, OPTIONAL MATCH, RETURN, WITH, WHERE, and ORDER BY (sub)clauses (all of them)
- Aggregation in at least one query
- Requirements:
- Describe the meaning of your Cypher expressions in natural language (via // comment)
- Submission:
- queries.cypher: text file with a sequence of Cypher statements (including CREATE)
- Execution:
- Execute the following shell command to evaluate the whole Neo4j script
- cat $ScriptFile | neo4j-shell --path $DatabasePath --non-interactive
- $DatabasePath is a path to a directory that should contain / already contains Neo4j embedded database to be used, e.g. /home/r192_login/MyNeo4jDatabase
- $ScriptFile is a file with Cypher queries to be executed, i.e. queries.cypher
- Tools:
- References:
- Server: nosql.ms.mff.cuni.cz:42222
- Do not forget to execute the homework submission script!
- Deadline: Sunday 21. 6. 2020 until 23:59
Individual Topics
- Try to propose your own original topic in the first place
- You can also get inspired by the following topics (in alphabetical order)
-
Access system,
Accommodation booking,
Accommodation comparator,
Accommodation sharing,
Agricultural production,
Air rescue service,
Air traffic management,
Airline,
Airport,
Armory,
Army,
Artworks,
Assignment submission,
ATM network,
Attendance system,
Auction,
Bakery,
Bank,
Bank account,
Bazaar,
Beekeeper,
Betting shop,
Beverages store,
Bike sharing,
Blog,
Boat rental,
Bookstore,
Botanic garden,
Brewery,
Bus station,
Bus tickets,
Business register,
Cadastre,
Cafe,
Canteens,
Car rental,
Car repair shop,
Casino,
Castles,
Catering,
Caves,
Cemetery,
Cinema,
City tours,
Classbook,
Collection and disposal of waste,
Collection of laws,
College dorm,
Computer games,
Conference,
Construction management,
Content management system,
Contract register,
Cookbook,
Cooking classes,
Council meetings,
Countries of the world,
Courier service,
Cowshed,
Dance school,
Deliveries,
Desk games,
Discussion forum,
Doctor's office,
Dog park,
Dog shelter,
Driving school,
Drugs,
Dump,
Educational institution,
Elections,
Electronic prescriptions,
Employee records,
Entertainment center,
Environmental center,
Exhibition,
Exhibition grounds,
Experience donation,
Fairy tales,
Farmer markets,
Finance manager,
Financial advisory,
Financial markets,
Fire protection,
Fishing equipment,
Fitness center,
Fleet,
Flight ticket booking,
Food bank,
Food distribution,
Football league,
Football team,
Forest kindergarten,
Foster care,
Gallery,
Gardening colony,
Gas station,
Glassworks,
Golf clubs,
Grant agency,
Grid,
Hair salon,
Handyman,
Hardware,
Health insurance,
High school,
Highway fees,
Hiking trails,
Hobby market,
Hockey league,
Holiday offers,
Horse racing,
Hospital,
Hotel,
Chamber of deputies,
Chess database,
Incinerator,
Industrial zone,
Insurance company,
Intelligence service,
Intersport arena,
Job offers,
Jurassic park,
Kindergarten,
Laboratory,
Labour office,
Language school,
Lego,
Leisure activities,
Library,
Log book,
Logistics center,
Logistics company,
Logistics warehouse,
Lottery,
Luggage storage,
Manufacturing processes,
Maternity hospital,
Medical reimbursement,
Meeting scheduling,
Menu,
Metro operation,
Military area,
Mobile operator,
Mobile phones,
Model trains,
Morgue,
Mountain rescue service,
Movies,
Multinational company,
Multiplex network,
Museum,
Music festival,
Music production,
Musical instruments,
National parks,
Nature reserve,
Newspaper publishing,
Non-bank loans,
Nuclear power plant,
Nutritional values,
Online exercises,
Online streaming service,
Orienteering,
Parking lot,
Parts catalog,
Patient medical card,
Pawnshop,
Payment cards,
Personal documents,
Personal trainer,
Pharmacy,
Photo album,
Pizzeria,
Plagiarism detection,
Planning calendar,
Police database,
Political parties,
Popular music,
Population register,
Post,
Postal addresses,
Poultry farming,
Prison,
Procurement,
Project management,
Property administration,
Psychiatric hospital,
Public greenery,
Public transport,
Railway network,
Real estate agency,
Recruitment agency,
Refugee camp,
Regulatory fees,
Research projects,
Research publications,
Restaurant,
Restaurant reservations,
Road closures,
Room reservation,
Scout group,
Scrapyard,
Security agency,
Seizures,
Shared travel,
Shooting range,
Shopping center,
Ski school,
Skiing area,
Social benefits,
Social network,
Software development,
Spare parts,
Sports club,
Sports tournament,
Stable,
Statement of work,
Stock exchange,
Student book,
Study abroad,
Study materials,
Study system,
Subsidy programs,
Summer camp,
Supermarket,
Sweet-shop,
Swimming pool,
Symphony orchestra,
Tax office,
Taxi service,
Teahouse,
Theater,
Theater plays,
Time tables,
Tollgates,
Tourism,
Tourist group,
Traffic accidents,
Traffic control center,
Train station,
Transport company,
Transport control,
Travel agency,
Trial,
Truck transport,
TV program,
TV series,
Universe,
Vaccination abroad,
Veterinary clinic,
Video shop,
Virtual tours,
Visas,
War conflicts,
Water park,
Water supply,
Weapons,
Weather forecast,
Webhosting,
Webshop,
Wedding dress rental,
Wholesale,
Winter road cleaning,
World heritage list,
Zoning plan,
Zoo
- Nevertheless, the following topics are not allowed this semester
- Movies, actors
- Airport, flights
Exam Requirements
NoSQL Introduction
- Big Data and NoSQL terms, V characteristics (volume, variety, velocity, veracity, value, validity, volatility), current trends and challenges (Big Data, Big Users, processing paradigms, ...), principles of relational databases (functional dependencies, normal forms, transactions, ACID properties); types of NoSQL systems (key-value, wide column, document, graph, ...), their data models, features and use cases; common features of NoSQL systems (aggregates, schemalessness, scaling, flexibility, sharding, replication, automated maintenance, eventual consistency, ...)
Data Formats
- XML: constructs (element, attribute, text, ...), content model (empty, text, elements, mixed), entities, well-formedness; document and data oriented XML
- JSON: constructs (object, array, value), types of values (strings, numbers, ...); BSON: document structure (elements, type selectors, property names and values)
- RDF: data model (resources, referents, values), triples (subject, predicate, object), statements, blank nodes, IRI identifiers, literals (types, language tags); graph representation (vertices, edges); N-Triples notation (RDF file, statements, triple components, literals, IRI references); Turtle notation (TTL file, prefix definitions, triples, object and predicate-object lists, blank nodes, prefixed names, literals)
- CSV: constructs (document, header, record, field)
- Protocol Buffers: components (interface description language, source code compiler, binary serialization format), intended usage; schema structure (messages, enumerations), message fields (rules, types, names, tags)
XML Databases
- Native XML databases vs. XML-enabled relational databases; data model (XDM): tree (nodes for document, elements, attributes, texts, ...), document order, reverse document order, sequences, atomic values, singleton sequences
- XPath language: path expressions (relative vs. absolute, evaluation algorithm), path step (axis, node test, predicates), axes (forward: child, descendant, following, ...; reverse: parent, ancestor, preceding, ...; attribute), node tests, predicates (path conditions, position testing, ...), abbreviations
- XQuery language: path expressions, direct constructors (elements, attributes, nested queries, well-formedness), computed constructors (dynamic names), FLWOR expressions (for, let, where, order by, and return clauses), typical FLWOR use cases (joining, grouping, aggregation, integration, ...), conditional expressions (if, then, else), switch expressions (case, default, return), universal and existential quantified expressions (some, every, satisfies), comparisons (value, general, node; errors), atomization of values (elements, attributes)
RDF Stores
- Linked Data: principles (identification, standard formats, interlinking, open license), Linked Open Data Cloud
- SPARQL: graph pattern matching (solution sequence, solution, variable binding, compatibility of solutions), graph patterns (basic, group, optional, alternative, graph, minus); prologue declarations (BASE, PREFIX clauses), SELECT queries (SELECT, FROM, and WHERE clauses), query dataset (default graph, named graphs), variable assignments (BIND), FILTER constraints (comparisons, logical connectives, accessors, tests, ...), solution modifiers (DISTINCT, REDUCED; aggregation: GROUP BY, HAVING; sorting: ORDER BY, LIMIT, OFFSET), query forms (SELECT, ASK, DESCRIBE, CONSTRUCT)
MapReduce
- Programming models, paradigms and languages; parallel programming models, process interaction (shared memory, message passing, implicit interaction), problem decomposition (task parallelism, data parallelism, implicit parallelism)
- MapReduce: programming model (data parallelism, map and reduce functions), cluster architecture (master, workers, message passing, data distribution), map and reduce functions (input arguments, emission and reduction of intermediate key-value pairs, final output), data flow phases (mapping, shuffling, reducing), input parsing (input file, split, record), execution steps (parsing, mapping, partitioning, combining, merging, reducing), combine function (commutativity, associativity), additional functions (input reader, partition, compare, output writer), implementation details (counters, fault tolerance, stragglers, task granularity), usage patterns (aggregation, grouping, querying, sorting, ...)
- Apache Hadoop: modules (Common, HDFS, YARN, MapReduce), related projects (Cassandra, HBase, ...); HDFS: data model (hierarchical namespace, directories, files, blocks, permissions), architecture (NameNode, DataNode, HeartBeat messages, failures), replica placement (rack-aware strategy), FsImage (namespace, mapping of blocks, system properties) and EditLog structures, FS commands (ls, mkdir, ...); MapReduce: architecture (JobTracker, TaskTracker), job implementation (Configuration; Mapper, Reducer, and Combiner classes; Context, write method; Writable and WritableComparable interfaces), job execution schema
NoSQL Principles
- Scaling: scalability definition; vertical scaling (scaling up/down), pros and cons (performance limits, higher costs, vendor lock-in, ...); horizontal scaling (scaling out/in), pros and cons, network fallacies (reliability, latency, bandwidth, security, ...), cluster architecture; design questions (scalability, availability, consistency, latency, durability, resilience)
- Distribution models: sharding: idea, motivation, objectives (balanced distribution, workload, ...), strategies (mapping structures, general rules), difficulties (evaluation of requests, changing cluster structure, obsolete or incomplete knowledge, network partitioning, ...); replication: idea, motivation, objectives, replication factor, architectures (master-slave and peer-to-peer), internal details (handling of read and write requests, consistency issues, failure recovery), replica placement strategies; mutual combinations of sharding and replication
- CAP theorem: CAP guarantees (consistency, availability, partition tolerance), CAP theorem, consequences (CA, CP and AP systems), consistency-availability spectrum, ACID properties (atomicity, consistency, isolation, durability), BASE properties (basically available, soft state, eventual consistency)
- Consistency: strong vs. eventual consistency; write consistency (write-write conflict, context, pessimistic and optimistic strategies), read consistency (read-write conflict, context, inconsistency window, session consistency), read and write quora (formulae, motivation, workload balancing)
Key-Value Stores
- Data model (key-value pairs), key management (real-world identifiers, automatically generated, structured keys, prefixes), basic CRUD operations, use cases, representatives, extended functionality (MapReduce, TTL, links, structured store, ...)
- Riak: data model (buckets, objects, metadata headers); HTTP interface, cURL tool (options); CRUD operations (POST, PUT, GET, and DELETE methods, structure of URLs, data, headers), buckets operations (buckets, keys, properties); links (definition, headers, tags, link walking, navigational steps: bucket, tag and keep components), data types (Convergent Replicated Data Types: register, flag, counter, set, map; conflict resolution policies; usage restrictions), Search 2.0 Yokozuna (architecture; indexation and query evaluation processes; extractors: text, XML, JSON; SOLR document: extracted and technical fields; indexing schema: tokens, triples; full-text index creation, association and usage; query patterns: wildcards, ranges, ...); causal context (motivation, low-level techniques: timestamps, vector clocks, ...); vector clocks (logical clocks, vector of clocks, message passing); Riak Ring (physical vs. virtual nodes, consistent hashing, partitions, replica placement strategy, hinted handoff, handling of read and write requests)
- Redis: features (in-memory, data structure store), data model (databases, objects), data types (string, list, set, sorted set, hash), string commands (SET, GET, APPEND, SETRANGE, INCR, DEL, ...), list commands (LPUSH, RPUSH, LPOP, RPOP, LINDEX, LRANGE, LREM, ...), set commands (SADD, SISMEMBER, SUNION, SINTER, SDIFF, SREM, ...), sorted set commands (ZADD, ZRANGE, ZRANGEBYSCORE, ZINCRBY, ZREM, ...), hash commands (HSET, HMSET, HGET, HMGET, HKEYS, HVALS, HDEL, ...), general commands (EXISTS, KEYS, DEL, RENAME, ...), time-to-live commands (EXPIRE, TTL, PERSIST)
Wide Column Stores
- Data model (column families, rows, columns), query patterns, use cases, representatives
- Cassandra: data model (keyspaces, tables, rows, columns), primary keys (partition key, clustering columns), column values (missing; empty; native data types, tuples, user-defined types; collections: lists, sets, maps; frozen mode), additional data (TTL, timestamp); CQL language: DDL statements: CREATE KEYSPACE (replication options), DROP KEYSPACE, USE keyspace, CREATE TABLE (column definitions, usage of types, primary key), DROP TABLE, TRUNCATE TABLE; native data types (int, varint, double, boolean, text, timestamp, ...); literals (atomic, collections, ...); DML statements: SELECT statements (SELECT, FROM, WHERE, GROUP BY, ORDER BY, and LIMIT clauses; DISTINCT modifier; selectors; non/filtering queries, ALLOW FILTERING mode; filtering relations; aggregates; restrictions on sorting and aggregation), INSERT statements (update parameters: TTL, TIMESTAMP), UPDATE statements (assignments; modification of collections: additions, removals), DELETE statements (deletion of rows, removal of columns, removal of items from collections)
Document Stores
- Data model (documents), query patterns, use cases, representatives
- MongoDB: data model (databases, collections, documents, field names), document identifiers (features, ObjectId), data modeling (embedded documents, references); CRUD operations (insert, update, save, remove, find); insert operation (management of identifiers); update operation: replace vs. update mode, multi option, upsert mode, update operators (field: $set, $rename, $inc, ...; array: $push, $pop, ...); save operation (insert vs. replace mode); remove operation (justOne option); find operation: query conditions (value equality vs. query operators), query operators (comparison: $eq, $ne, ...; element: $exists; evaluation: $regex, ...; logical: $and, $or, $not; array: $all, $elemMatch, ...), dot notation (embedded fields, array items), querying of arrays, projection (positive, negative), projection operators (array: $slice, $elemMatch), modifiers (sort, skip, limit); MapReduce (map function, reduce function, options: query, sort, limit, out); primary and secondary index structures (index types: value, hashed, ...; forms; properties: unique, partial, sparse, TTL)
Graph Databases
- Data model (property graphs), use cases, representatives
- Neo4j: data model (graph, nodes, relationships, directions, labels, types, properties), properties (fields, atomic values, arrays); embedded database mode; traversal framework: traversal description, order (breadth-first, depth-first, branch ordering policies), expanders (relationship types, directions), uniqueness (NODE_GLOBAL, RELATIONSHIP_GLOBAL, ...), evaluators (INCLUDE/EXCLUDE and CONTINUE/PRUNE results; predefined evaluators: all, excludeStartPosition, ...; custom evaluators: evaluate method), traverser (starting nodes, iteration modes: paths, end nodes, last relationships); Java interface (labels, types, nodes, relationships, properties, transactions); Cypher language: graph matching (solutions, variable bindings); query sub/clauses (read, write, general); path patterns, node patterns (variable, labels, properties), relationship patterns (variable, types, properties, variable length); MATCH clause (path patterns, WHERE conditions, uniqueness requirement, OPTIONAL mode); RETURN clause (DISTINCT modifier, ORDER BY, LIMIT, SKIP subclauses, aggregation); WITH clause (motivation, subclauses); write clauses: CREATE, DELETE (DETACH mode), SET (properties, labels), REMOVE (properties, labels); query structure (chaining of clauses, query parts, restrictions)
Recommended Literature
- Sadalage, Pramod J. - Fowler, Martin: NoSQL Distilled.
ISBN: 978-0-321-82662-6.
Pearson Education, Inc., 2013.
- Wiese, Lena: Advanced Data Management: For SQL, NoSQL, Cloud and Distributed Databases.
ISBN: 978-3-11-044140-6 (hardcover), 978-3-11-044141-3 (eBook PDF), 978-3-11-043307-4 (eBook EPUB).
DOI: 10.1515/9783110441413.
Walter de Gruyter GmbH, 2015.
- Zomaya, Albert Y. - Sakr, Sherif: Handbook of Big Data Technologies.
ISBN: 978-3-319-49339-8 (hardcover), 978-3-319-49340-4 (eBook).
DOI: 10.1007/978-3-319-49340-4.
Springer International Publishing AG, 2017.