Correct, it's only intended to match the first file in the torrent.

Correct.

Oh, I see the problem now. Thanks for your explanation. The last block of the file might contain information from the next file, and since that would be included in the block hash, it wouldn't be useful for detecting duplicates.

My thought process was to split the file into blocks and hash the partial last block, but I didn't consider the interaction with torrents containing multiple files.

Sorry to be a bother, but if I understand correctly, this would also not allow for detecting duplicate files that aren't aligned to block boundaries, is that the case?

It would also be nice to know the rich text formatting used on this feedback page. Figured it out, it's pseudo-HTML.

Another minor question: the documentation says "CRC32" a lot sometimes, which CRC polynomial is actually used here? The Ethernet one?

As opposed to including the partial block at the end? If so, I don't see how you could make it work.

Keep in mind that the point is to provide an indicator to detect duplicates. Arguably it's not 100% accurate, but it should be close.
You don't know how the partial block at the end will be filled - if it's the last block of the torrent, it isn't padded, if it's not the last block, it's filled with the contents of the next file. There's no way to compute the hash for the latter case, since you can't anticipate what file will be concatenated onto the end. The former case is computable, but would only be useful to match against torrents where that file is at the end and the file's offset is block aligned - basically it'd only be useful for matching against single file torrents, but useless for matching batches.

(Does this use BBCode?)

[quote]The hash isn't intended to detect errors, but if you used it for such, then yes.[/quote]
Ah, of course, my bad. Ditto for duplicate detection though, files with different trailing data will hash equal. Is there a reason it's done this way? In any case, it's not like it can be changed at this point, and even if, it would be of small benefit, if any. Thanks for the quick response.

The hash isn't intended to detect errors, but if you used it for such, then yes.

It's just a raw LZMA2 stream with a preset dictionary and CRC32 checksum.

Thank you for the compliments.

Also, your SIMD stuff is pretty nice.

From the DB export page:

> torpc_sha1_*: hex encoded SHA1 hash of concatenated SHA1 hashes (binary encoded) of the respective block size. For example, the torpc_sha1_16k hash is obtained by breaking the file into 16KB blocks (if the last block is less than 16KB, it is discarded), calculating a 20 byte SHA1 hash for each block, concatenating these hashes, which is then fed through SHA1 to obtain the final hash. The selected block sizes correspond with the most common piece sizes used for torrents, and hence this hash can be useful in trying to detect duplicate torrents which have different info hash values.

This means that errors in the last block will not be detected, right?

> Mediainfo and related data are currently not included mainly due to size and time it takes to dump the data. The data is also compressed using a custom LZMA2 based scheme, which users would need to implement a decompressor for. I may consider including this data if many are interested in such.

This has made me curious about that scheme. Would you mind sharing it or some code implementing it?

Thanks for the awesome site, cute design, and making these database dumps available :) Very cool!