Recommending Frameworks - cont.

Custom code in frameworks

While last time we focused on out-of-the-box functionality in the recommending frameworks, this time we will explore a bit how convenient it is to actually modify some portions of the framework.

• Note that the main task is not to have this completed, but rather to gain insight on the framework's usability.
• Try to maintain the notes and then fill-in the GoogleDoc: https://docs.google.com/spreadsheets/d/1Bpb4TkW0teDQvwKryswh_Fhrt2qa-cqAfwuPRx-oxSo/edit?usp=sharing

Task 1: select which framework you wish to work with

• keep the one from Lab1, or select any other

Task 2: custom-made evaluation of recommending algorithms

How difficult it will be to made a custom evaluation metric and integrate it into the framework. For this task, use some dataset with genre information available (e.g., MovieLens, LibraryThing, GoodBooks etc.). Apart from standard evaluation metrics, you would like to check a custom-made one as well. How difficult is this in your selected framework?

• Consider "genre-wise serendipity" as your target. Serendipity aims to measure how many recommendations were both relevant and surprising. For genre-wise serendipity, we gonna define the "surprisingness" through genres. In particular, the item is surprising, if and only if its genres are not present in the genres of the existing user profile (i.e., there is no rated item with the same genre in the users train set). Relevance can be considered as an existence of the recommended item in the test set (you can also apply a filter on the numerical rating if you want to). Furthermore, you want this metric to be defined in a recall fashion - i.e., how many of the potentially serendipitous recommendations (from the user profile) the algorithm actually recommended?

  • $Serendipity_G := \frac{\sum_{i \in RecList} Surprising(i,U) \ast Relevant(i,U)}{ |Surprising\_and\_Relevant(U)|}$
  • $Surprising(i,U)$ is True if $\forall g \in i, \forall j \in trainset(U): g \notin j $
  • $Relevant(i,U)$ is True if $i \in testset(U)$
  • $Surprising\_and\_Relevant(U)$ returns all items that are present in the test set and are surprising (w.r.t. genre) for given user U

• Implement the metric, incorporate it into the framework and use it in some evaluation scenario (e.g., compare two algorithms, or several hyperparameters of one w.r.t. genre-wise serendipity)

Notes

• the framework have to be able to handle item's metadata
• per-user test-set needs to be linked to item's metadata
• probably, you gonna need to inherit from some generic eval metric + this has to have some suitable interface
• in ideal case, you might want to filter the test users to only those that has non-empty results in $Surprising\_and\_Relevant(U)$
• Highlight to what extent you can do this in line with how the framework was designed vs. whether some dirty trics are necessary

First outline the concept of the solution, then try to implement it

[If enough time] Task 3: custom-made recommending algorithm

Probably the most common thing you may need from RS framework is to test your own algorithm there. How difficult would that be in your framework? Another notorious issue is the usage of additional data beyond user feedback within the frameworks. Therefore, we gonna focus on content-based RS. Consider a simple Item KNN working on top of content-based similarity - an example of such is https://github.com/yjeong5126/movie_recommender/blob/master/content_based_filtering/content_based_recommender.ipynb.

• Check how the framework handles content-based metadata of items (and how difficult is to use additional dataset).
  • How to add new algorithm (subclassing / registering / config update / ...?)
  • How to work with item metadata (data joining,...)?
• Adapt the existing implementation for usage in the framework.