Tag Archives: givealink


Web Science 2014 Data Challenge

The datasets described below are used in the Web Science 2014 Data Challenge. For more, information, please the call for participation. For updates, see the Data Challenge section of the Web Science 2014 website.

There are 4 datasets in this collection. Each is available as a .tar.gz file containing either .json or .csv files. When the JSON format is used, each .json file contains a single JSON object. The format of that object is dependent on the dataset. See below for details. The datasets have been prepared by Dimitar Nikolov.

1. Web Traffic

A collection of Web (HTTP) requests for the month of November 2009. This is a small sample of the larger click dataset, documented here. (More on Web Traffic project).

JSON object format:

    'timestamp': 123456789, # Unix timestamp
    'from': '...', # the referrer host
    'to': '...', # the target host
    'count': 1234 # the number of request between the referrer and target hosts that occurred within the given hour

The data has been aggregated for every hour of the day. Thus, if more than one request occurred from the same referrer host to the same target host between, say, 2pm and 3pm, this is reflected in the ‘count’ field of the JSON object with a timestamp for 2pm, rather than by a different JSON object with a different timestamp.

Dataset statistics:

  • Dataset size: 235M requests
  • File size: 2.7GB uncompressed
  • Time period: Nov 1, 2009 – Nov 22, 2009

Data: web-clicks-nov-2009.tgz (321MB)

If you use this dataset in your research, please cite either or both of these papers:

    title = {Ranking Web Sites with Real User Traffic},
    author = {Meiss, M. and Menczer, F. and Fortunato, S. and Flammini, A. and Vespignani, A.},
    booktitle = {Proc. First ACM International Conference on Web Search and Data Mining (WSDM)},
    url = {http://informatics.indiana.edu/fil/Papers/click.pdf},
    pages = {65--75},
    year = 2008
    title = {Modeling Traffic on the Web Graph},
    author = {Meiss, M. and Goncalves, B. and Ramasco, J. and Flammini, A. and Menczer, F.},
    booktitle = {Proc. 7th Workshop on Algorithms and Models for the Web Graph (WAW)},
    series = {Lecture Notes in Computer Science},
    url = {http://informatics.indiana.edu/fil/Papers/abc.pdf},
    pages = {50--61},
    volume = 6516,
    year = 2010


2. Twitter

A collection of records extracted from tweets for the month of November 2012 containing both #hashtags and URLs as part of the tweet. (More on Truthy project)

JSON object format:

    'timestamp': 123456789, # Unix timestamp
    'user_id': 12345, # an integer uniquely identifying the user who tweeted
    'hashtags': ['...', '...', '...'], # a list of hashtags used in the tweet
    'urls': ['...', '...', '...'] # a list of links used in the tweet

Dataset statistics:

  • Dataset size: 27.8M tweets
  • File size: 3.5GB uncompressed
  • Time Period: Nov 1, 2012 – Nov 30, 2012

Data: tweets-nov-2012.json.gz (865MB)

If you use this dataset in your research, please cite either or both of these papers:

    author = {McKelvey, Karissa and Menczer, Filippo},
    title = {Design and prototyping of a social media observatory},
    booktitle = {Proceedings of the 22nd international conference on World Wide Web companion},
    series = {WWW '13 Companion},
    pages = {1351--1358},
    url = {http://dl.acm.org/citation.cfm?id=2487788.2488174},
    year = 2013
    Author = {Karissa McKelvey and Filippo Menczer},
    Title = {{Truthy: Enabling the Study of Online Social Networks}},
    Booktitle = {Proc. 16th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (CSCW)},
    Url = {http://arxiv.org/abs/1212.4565},
    Year = 2013


3. Social Bookmarking

A collection of bookmarks from GiveALink.org for the month of November 2009. (More on GiveALink project)

JSON object format:

    'timestamp': 123456789, # Unix timestamp for when the URL was posted
    'url': '...', # the URL that was bookmarked
    'hashtags': ['...', '...', '...'] # a set of tags attached to the URL by the (anonymous) user

Dataset statistics:

  • Dataset size: 61,665 posts (approximately 430,000 triples)
  • File size: 12MB uncompressed
  • Time period: Nov 1, 2009 – Nov 30, 2009

Data: givealink-nov-2009.tgz (2MB)

If you use this dataset in your research, please cite either or both of these papers:

    author = {Markines, B. and Stoilova, L. and Menczer, F.},
    title = {Bookmark hierarchies and collaborative recommendation},
    booktitle = {Proc. 21st National Conference on Artificial Intelligence (AAAI-06)},
    pages = {1375--1380},
    publisher = {AAAI Press},
    url = {http://www.aaai.org/Papers/AAAI/2006/AAAI06-216.pdf},
    year = 2006
    Author = {Stoilova, Lubomira and Holloway, Todd and Markines, Ben and Maguitman, Ana G. and Menczer, Filippo},
    Title = {GiveALink: Mining a Semantic Network of Bookmarks for Web Search and Recommendation},
    Booktitle = {Proc. KDD Workshop on Link Discovery: Issues, Approaches and Applications (LinkKDD)},
    Url = {http://informatics.indiana.edu/fil/Papers/givealink-linkkdd.pdf},
    Year = 2005


4. Publications

Metadata for the complete set of all PubMed records through 2012 (with part of 2013 available as well), including title, authors, and year of publication. All data provided originates from NLM’s PubMed database (as downloaded April 24, 2013 from the NLM FTP site) and was retrieved via the Scholarly Database.

CSV format:

PubMed ID1,title1,year of publication1,author1|author2|author3|…
PubMed ID2,title2,year of publication2,author4|author1|author5|…

Dataset statistics:

  • Dataset size: 21.5 mil publications and 10.8 mil authors
  • File size: 3.1GB uncompressed
  • Time period: 1809 – 2013

Data: publications-1809-2013.tar.gz (1.4GB)

If you use this dataset in your research, please cite either or both of these papers:

  author    = {Light, Robert P., David E. Polley and Katy Börner},
  title     = {Open Data and Open Code for Big Science of Science Studies},
  booktitle = {Proceedings of International Society of Scientometrics and Informetrics Conference},
  year      = {2013},
  pages     = {1342--1356},
  url       = {http://cns.iu.edu/docs/publications/2013-light-sdb-sci2-issi.pdf}
  author  = {Rowe, Gavin La, Sumeet Adinath Ambre, John W. Burgoon, Weimao Ke, and Katy Börner},
  title   = {The Scholarly Database and its Utility for Scientometrics Research"},
  journal = {Scientometrics},
  year   = {2009},
  volume = {79},
  number = {2},
  month  = {May},
  url    = {http://cns.iu.edu/docs/publications/2009-larowe-sdb.pdf}

NaN’s strong presence at HT09

ht09NaN had a strong presence at Hypertext 2009 in Torino:

Summer talks in Europe

I gave four invited talks in Spain, Italy, and Switzerland this summer:

Thanks to my wonderful hosts and their groups for engaging discussions and delightful  hospitality!


givealinkUPDATE: As of 2015 the GiveAlink project has been archived and the GiveALink.org website is no longer operational.

Link analysis algorithms leverage hyperlinks created by authors as semantic endorsements between pages, while social bookmarks provide a way to leverage annotations by information consumers as a source of information about pages. This project explores a novel approach that is a synergy of the two: soliciting annotations from users about the content of pages, in a way that implicitly forms networks of relationships between and among resources and tags. These socially generated relationships are then aggregated to build bottom-up, global semantic similarity networks. Algorithms are developed to construct, analyze, and mine these networks in support of search and recommendation applications, exploratory navigation interfaces, resource management utilities, tag spam detection, and incentive games to accelerate the achievement of critical mass.

To extrapolate both annotations about content (tags) and semantic relationships (similarity) from single users to the “wisdom of the crowd,” the project investigates an information-theoretic model that extracts semantic assessments from information structures that many users are already maintaining, namely the bookmarks and tags they manage on their browsers or online. This entails the design and evaluation of several network-based measures and algorithms, such as similarity, novelty, centrality, and focus. Among the aims of this model are the exploration of the duality between resources (URLs) and concepts (tags or categories) and the integration of social annotation and collaborative filtering. One way to provide users with immediate value is to integrate client-based taxonomies and server-based folksonomies for social bookmark management. Both traditional users of browser bookmarks and social users of online bookmarks can take advantage of the same semantic maps while retaining the convenience of intuitive browser interfaces and centralized storage.

Strategic collaborations to share data, accelerate evaluation, and maximize impact are under way with key groups in Europe through the TAGora Project and its partners at Rome Sapienza, Sony Paris, the ISI Foundation in Torino, and the BibSonomy group at Kassel University. GiveALink.org (supported by a wonderful computing and storage infrastructure) is an open social bookmarking platform developed to experiment with and demonstrate the ideas of this project. The algorithms and data generated by the project are made available to the Web community to facilitate analysis, the development of improved network algorithms, and integration with other Internet applications. Early results of this project have been presented at various conferences and workshops including LinkKDD2005, AAAI2006, and HT2008. More recent publications are listed below. To learn more, donate your bookmarks, play with our system, and download our data and applications please visit GiveALink.org.

Project Members

Fil Menczer, PI
Fil Menczer (PI)
Lilian Weng
Dimitar Nikolov

Collaborators & Alumni:

Rossano Schifanella
Rossano Schifanella
Jacob Ratkiewicz
Jacob Ratkiewicz
Heather Roinestad
Heather Roinestad
Ben Markines
Ben Markines
ciro cattuto
Ciro Cattuto
Katrina Panovich
Katrina Panovich
Wouter Van den Broeck
Wouter Van den Broeck
John Burgoon
John Burgoon
Mira Stoilova
Mira Stoilova

We should also acknowledge Todd Holloway for his contributions to the early search engine; Luis Rocha and Ana Maguitman for suggesting the idea of ranking and searching by novelty; Mark Meiss, who thought of the catchy name for GiveALink; and Rob Henderson, quite possibly the greatest sysadmin around.


Related Publications

Wiki (team only)


This project is supported by the National Science Foundation under award IIS-0811994: Social Integration of Semantic Annotation Networks for Web Applications. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.


AIRWEB 2009 practice talk for Tuesday, 3/24

Dear NaNers,

I will be using Tuesday (3/24) as a practice talk for AIRWEB 2009. Hope everyone had a nice spring break!

Title: Social Spam Detection

The popularity of social bookmarking sites has made them prime targets for spammers. Many of these systems require an administrator’s time and energy to manually filter or remove spam. Here we discuss the motivations ofsocial spam, and present a study of automatic detection of spammers in a social tagging system. We identify and analyze six distinct features that address various properties of social spam, finding that each of these features provides for a helpful signal to discriminate spammers from legitimate users. These features are then used in various machine learning
algorithms for classification, achieving over 98% accuracy in detecting social spammers with 2% false positives. These promising results provide a new baseline for future efforts on social spam. We make our dataset publicly
available to the research community.


CSI Piemonte

No, it’s not an Italian spin-off of the popular TV show. CSI Piemonte is organizing a meeting on Understanding Complexity: a Journey through Science to be held November 22-23 at the Lingotto Convention Center here in Torino. We will have demos and posters on 6S, GiveALink, and the egalitarian effect of search engines. I look forward in particular to seeing my good old friend Dario and my mentor, Domenico.

4 talks at ECCS 2007

We are giving 4 invited talks at the European Conference on Complex Systems, in Dresden (October 1-5, 2008):
  1. Web click traffic network (by Fil) at Complex Networks: Dynamics and Topology Interplay