Negative Impact of Bots on Digital Repository Usage Statistics: A Case Analysis and Applied Strategy

• Autores: Bértoli, Rafael; Lira, Ariel Jorge
• Año de publicación: 2025
• Idioma: inglés
• Tipo de recurso: documento de conferencia
• Materia: Informática; bots; machine learning; repositories; usage statistics
• Institución: Universidad Nacional de La Plata

Bot detection in web services has been a necessity since the massification of the internet. These agents crawl the information available on the web for various purposes, such as search engine development, SEO analysis, or, recently, for training artificial intelligence models. Repositories are particularly interesting to these agents because they offer high-quality controlled information described by curated metadata. Regardless of its objective, the repository is continuously being harvested by multiple bots, which causes occasional downtime and alterations in the usage logs used for generating statistical reports, which serve to measure the real impact of the preserved and published works. This paper presents the experience of the SEDICI repository in analyzing and cleaning 13 years' worth of collected logs. Detection is performed by analyzing bot behavior, such as excessive usage, anomalous usage patterns, scans, and attacks. Then, an AI model is used to flag bots with subtle behavior not otherwise identified as such. By applying these strategies, it was possible to eliminate over 50 million usage records originating from atypical bots that systematically and recurrently access the repository.
Traducción al inglés de "Impacto negativo de bots en estadísticas de uso de repositorios digitales: análisis de un caso y estrategia aplicada" (ver "Documentos relacionados").
Dirección PREBI-SEDICI