Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Rewards2/Xcity_WebScraper/READ.MD
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Web scrapping?

Web scraping refers to the extraction of data from a website. This information is collected and then exported into a format that is more useful for the user. Be it a spreadsheet or an API.

# HOW DOES IT WORK

So, how do web scrapers work? Automated web scrapers work in a rather simple but also complex way. After all, websites are built for humans to understand, not machines.

First, the web scraper will be given one or more URLs to load before scraping. The scraper then loads the entire HTML code for the page in question. More advanced scrapers will render the entire website, including CSS and Javascript elements.

Then the scraper will either extract all the data on the page or specific data selected by the user before the project is run.

Ideally, the user will go through the process of selecting the specific data they want from the page. For example, you might want to scrape an Amazon product page for prices and models but are not necessarily interested in product reviews.
Lastly, the web scraper will output all the data that has been collected into a format that is more useful to the user.

Most web scrapers will output data to a CSV or Excel spreadsheet, while more advanced scrapers will support other formats such as JSON which can be used for an API.

# How to install

npm install[ to install modules]

[youtube]()
239 changes: 239 additions & 0 deletions Rewards2/Xcity_WebScraper/index.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
[
{
"close-rewards": 0,
"closing-amount": 0,
"confirmed-round": 62440,
"fee": 1000,
"first-valid": 62000,
"genesis-hash": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"genesis-id": "mainnet-v1.0",
"id": "7MK6WLKFBPC323ATSEKNEKUTQZ23TCCM75SJNSFAHEM65GYJ5ANQ",
"intra-round-offset": 0,
"last-valid": 63000,
"note": "QS4gOyBBbGV4IE1jQ2FiZSA7IEFsbGlzb24gTm9sYW4gOyBBbm5lIFdhcm5lciA7IGLimKVhIDsgQmVuamFtaW4gQ2hhbiA7IEJlbmphbWluIEQuIFdhcmQgOyBCbyBMaSA7IENocmlzIEh1cmxleSA7IGRlcmVrIDsgZGltaXRyaXMgOyBFbGxlIFlvdSA7IEVyaWMgR2llc2VrZSA7IEV2YW4gSmFtZXMgUmljaGFyZCA7IEdyZWcgQ29sdmluIDsgR1YgOyBIb2V0ZWNrIDsgSWFuIENyb3NzIDsgSmFrZSBFdmFuIEdyZWVuc3RlaW4gOyBKYXNvbiBXZWF0aGVyc2J5IDsgSmluZyBDaGVuIDsgS2FybWFzdGljIDsgS2F0cmljZSBHcmFkeSA7IEtlbGkgSiBDYWxsYWdoYW4gOyBMZW8gUmV5emluIDsgTGl6IEJhcmFuIDsgTWFrZW5hIFN0b25lIDsgTWFzb24oWXVkZSlIdWFuZyA7IE1hdXJpY2UgSGVybGloeSA7IE1heCBKdXN0aWN6IDsgTXVzcyA7IE5hdmVlZCBJaHNhbnVsbGFoIDsgTmlja29sYWkgWmVsZG92aWNoIDsgUGFibG8gQXphciA7IFBhdWwgUmllZ2xlIDsgUmVnaW5hIDsgcmZ1c3Rpbm8gOyBSb3RlbSBIZW1vICjXqNeV16rXnSDXl9ee15UpIDsgc2FtIGFiYmFzc2kgOyBTYXd5ZXIgSHVybGV5IDsgU2VyZ2V5IEdvcmJ1bm92ICjQodC10YDQs9C10Lkg0JPQvtGA0LHRg9C90L7QsikgOyBTaWx2aW8gTWljYWxpIDsgU3Jpaml0aCBQb2R1dmFsIDsgU3RldmVuIEtva2lub3MgOyBUc2FjaGkgSGVybWFuIDsgVHlsZXIgTWFya2xleSA7IFZpY3RvciBMdWNoYW5nY28gOyBXLiBTZWFuIEZvcmQgOyBXaWxsICJaZXJvTWljcm9uIiBXaW5kZXIgOyBZb3NzaSBHaWxhZCA7IFpoZW5mZWkgWmhhbmc=",
"payment-transaction": {
"amount": 100000,
"close-amount": 0,
"receiver": "ALGORANDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIN5DNAU"
},
"receiver-rewards": 0,
"round-time": 1560614017,
"sender": "I3345FUQQ2GRBHFZQPLYQQX5HJMMRZMABCHRLWV6RCJYC6OO4MOLEUBEGU",
"sender-rewards": 30576000000,
"signature": {
"multisig": {
"subsignature": [
{
"public-key": "fDPiywtmtrpA2WOY+Mx9y6etNBCij1VKwZmGWW4PbKk="
},
{
"public-key": "ETnffVmxyVfJtVgCWFuStLsPJna9G1SHA1yJrfIo6RU="
},
{
"public-key": "hYkIN+Iyt2675q+XuYwoAzwR8B0P17WTUFGYn456E4o=",
"signature": "eBLuSsmbqXTtKcoDpI88t7CNyQ7ggJ8ZMGjpy+hLWnvjNi938/5U6Eb25Dmes0WLkCxnDZG7gsj3YIDmZfFLAA=="
},
{
"public-key": "5ChQFEXiHWTeXoJCRymNn8rmEAJAxpaigu4wIgcaODU=",
"signature": "45ndEdxV115jUGBmqt4WSjcBDg847CiPlE0w5omziLftSRzOtJSd5zrF1zkHOa1B1GJV4AE8E2qriMIbifnYBw=="
},
{
"public-key": "RjQ91+zvYumrPm9UOEMN+GnlHW+0gliRCCV2b6KOlwk=",
"signature": "LbmMSdKaqD/s9M1ldNAvLYGRMwxWdVPbl4i2zBVKwRnrRLM1Ape9zWMAxX1yJGxk/mAKGa9lZwAfQUlyus58Cw=="
},
{
"public-key": "k5F6WQJGyeiPHaN7fvmnBXz6YNq4NQ6BguE7yUmRWkI=",
"signature": "47b3oXSW6ZVGXmnFy59iQZohcs79v4Da05MTNr0jkUAHl5kseS7Br0C838nbZB79Yj9+wt7kuiiJkCOFgAAwBw=="
}
],
"threshold": 4,
"version": 1
}
},
"tx-type": "pay"
},
{
"close-rewards": 0,
"closing-amount": 0,
"confirmed-round": 63974,
"fee": 1000,
"first-valid": 63000,
"genesis-hash": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"genesis-id": "mainnet-v1.0",
"id": "NOXQWFHT4ZPO7UKTQ2VYCQRTTH3LCV62K7PBRGW2Q6IVOHOUYRLA",
"intra-round-offset": 0,
"last-valid": 64000,
"note": "QW15IEZsZW1pbmcgOyBDcmFpZyBHZW50cnkgOyBGYWJyaWNlIEJlbmhhbW91ZGEgOyBGYW5nZmFuZyBDaGVuIDsgRmVkZXJpY28gRGVtaWNoZWxpIDsgSHVnbyBLcmF3Y3p5ayA7IEpvaG4gQnJpZGdld2F0ZXIgOyBNYXNzaW1vIE1vcmluaSA7IFNoYWkgSGFsZXZpIDsgVGFsIFJhYmlu",
"payment-transaction": {
"amount": 100000,
"close-amount": 0,
"receiver": "ALGORANDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIN5DNAU"
},
"receiver-rewards": 0,
"round-time": 1560620526,
"sender": "6LQH42A4QJ3Y27FGKJWERY3MD65SXM4QQCJJR2HRJYNB427IQ73YBI3YFY",
"sender-rewards": 31320000000,
"signature": {
"multisig": {
"subsignature": [
{
"public-key": "fDPiywtmtrpA2WOY+Mx9y6etNBCij1VKwZmGWW4PbKk="
},
{
"public-key": "ETnffVmxyVfJtVgCWFuStLsPJna9G1SHA1yJrfIo6RU="
},
{
"public-key": "hYkIN+Iyt2675q+XuYwoAzwR8B0P17WTUFGYn456E4o=",
"signature": "YGtwItEyWBXmaiGDSRU2s8PLdFgitwFbnriPbLZHSskaRDD3Z8bV1iIfhHVfu7Vt5t0uUefWr4icaLnXd3ZvAQ=="
},
{
"public-key": "RjQ91+zvYumrPm9UOEMN+GnlHW+0gliRCCV2b6KOlwk=",
"signature": "vBlpBpndXop7/1RAUNeU8sGmVakMmCcBtGe0zvA5bx2g/YDMbmc6AlcI+Mc/v6xor3bpwwUAkaa25IyYhP/FCA=="
},
{
"public-key": "5ChQFEXiHWTeXoJCRymNn8rmEAJAxpaigu4wIgcaODU=",
"signature": "zbPbXAWZ72WcFyKJVpzQjokmb9fYj0+9BQDDRZcXshYm40m4NwMnL+iUKzuquxKX6YxuK7YlrM5gx7Ma9wJSAQ=="
},
{
"public-key": "k5F6WQJGyeiPHaN7fvmnBXz6YNq4NQ6BguE7yUmRWkI=",
"signature": "YYXHxLlS+B9vPdjtGrUPYa3lVzbI7PGtJX1SM54E+RG6lMdHJ78/x64JoVCc/on9rtQbRKQuqGG1+nImfFBkAA=="
}
],
"threshold": 4,
"version": 1
}
},
"tx-type": "pay"
},
{
"close-rewards": 0,
"closing-amount": 0,
"confirmed-round": 64069,
"fee": 1000,
"first-valid": 64000,
"genesis-hash": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"genesis-id": "mainnet-v1.0",
"id": "J4EEXHF4D5SEVUEGP4ICND6VQODY5HZ4MOKKNC7VHB577IWPCWGA",
"intra-round-offset": 0,
"last-valid": 65000,
"note": "My4wIENhcGl0YWwgOyBBbGdvIENhcGl0YWwgYW5kIG90aGVycyA7IEFwcGxpZWQgQ3J5cHRvIFZlbnR1cmVzIDsgQXNzZXRCbG9jayA7IEJpeGluIENhcGl0YWwgOyBCcmFpbmNoaWxkIEhvbGRpbmdzIDsgQ2FyZGluYWwgUGl0Y2ggQ2x1YiBMaW1pdGVkIDsgQ2x1c3RlciBDYXBpdGFsIDsgQ01CIEludGVybmF0aW9uYWwgOyBDb2duaXRpdmUgQmxvY2tjaGFpbiA7IENvaW5GYWJyaWsgOyBDb2xsYWJvcmF0aXZlIEZ1bmQgOyBDb250aW51ZSBDYXBpdGFsIDsgRWxldmVuIEVsZXZlbiBJbnZlc3RtZW50cyA7IEV0ZXJuYSBDYXBpdGFsIDsgRmxpcHNpZGUgQ3J5cHRvIDsgRm91bmRhdGlvbiBDYXBpdGFsIDsgSGV4IENhcGl0YWwgOyBJTU8gVmVudHVyZXMgOyBKTiwgU0FULCBCUCwgU1MsIFJNLCBCUiwgQ0IsIERTLCBKUywgVEwsIExLLCBXQywgVkwsIFpTLCBCRiwgTUMsIE1PLCBNRiwgUEEsIEdXLCBDSCwgSkcsIFNMLCBVQSwgTVZEUyA7IEtvc21vcyA7IExlbW5pc2NhcCA7IEx2bmEgQ2FwaXRhbCA7IE1ldGFTdGFibGUgQ2FwaXRhbCA7IE1FV04gOyBNb3phaWsgQ2FwaXRhbCA7IE11bHRpY29pbiBDYXBpdGFsIDsgTmVrdHJhIFMuQS4gOyBORU8gR2xvYmFsIENhcGl0YWwgOyBOZXdTdHlsZSBDYXBpdGFsIDsgTkdDIFZlbnR1cmVzIDsgTmlydmFuYSBDYXBpdGFsIDsgUDJQIENhcGl0YWwgOyBQaWxsYXIgOyBQb2x5Yml1cyBDYXBpdGFsIDsgUHVyZVN0YWtlIC0gRGVyZWssIERldm9uLCBKb2UsIE9za2FyLCBTdGVmYW4sIFRpbSA7IFFDUCBDYXBpdGFsIDsgUUNQIFNvdGVyaWEgOyBTbG93IFZlbnR1cmVzIE9mZnNob3JlIENyeXB0byA7IFNwYXJrIFBvb2wgOyBTdGFrZWQgQ2F5bWFuIDsgVHJhbnNmZXJlbmNlIEZ1bmQgOyBVbmlvbiBTcXVhcmUgVmVudHVyZXMgOyBWTEFORSBDQVBJVEFMIDsgV2FsdGVyIFQuRS4gRGFuY28gOyBXaW5nIFZDIDsgWW91YmkgQ2FwaXRhbCA7IGFuZCBtb3Jl",
"payment-transaction": {
"amount": 100000,
"close-amount": 0,
"receiver": "ALGORANDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIN5DNAU"
},
"receiver-rewards": 0,
"round-time": 1560620929,
"sender": "3V2MC7WJGAFU2EHWBHEETIMJVFJNAT4KKWVPOMJFJIM6ZPWEJRJ4POTXGI",
"sender-rewards": 31368000000,
"signature": {
"multisig": {
"subsignature": [
{
"public-key": "fDPiywtmtrpA2WOY+Mx9y6etNBCij1VKwZmGWW4PbKk="
},
{
"public-key": "ETnffVmxyVfJtVgCWFuStLsPJna9G1SHA1yJrfIo6RU="
},
{
"public-key": "5ChQFEXiHWTeXoJCRymNn8rmEAJAxpaigu4wIgcaODU=",
"signature": "cQj9Bo7KTrznj19gOAFj4NhOdpxY0AEckKl1RqjHCEargWy1j4qzCBQ5iK+nSxreFpUxdgqjulbljoqyZ/muDg=="
},
{
"public-key": "hYkIN+Iyt2675q+XuYwoAzwR8B0P17WTUFGYn456E4o=",
"signature": "J55Qv4y4c5D2lX9pHz3nN5OCDmV831nEp4aZgTPoab+HXZWqbH1FKGPp5MXN9gdgxseV4qR5jXEewi7IjR3tAQ=="
},
{
"public-key": "RjQ91+zvYumrPm9UOEMN+GnlHW+0gliRCCV2b6KOlwk=",
"signature": "32S+jCCV+pYKdbvzQJdW4fWfM70hWLfBR5/UxXvVn2m4N9pVxwKyI9CqdX7VcpeB3K52utOkWVLZrC4UhykjAg=="
},
{
"public-key": "k5F6WQJGyeiPHaN7fvmnBXz6YNq4NQ6BguE7yUmRWkI=",
"signature": "K+tbAqjkvg6I3gHv9Nl6MXXvXyuGHHOzMg4ltyhAruZeQSW/KcreSGo6O2033CSk1hy1htlDWxiZFvlXJIrMDA=="
}
],
"threshold": 4,
"version": 1
}
},
"tx-type": "pay"
},
{
"close-rewards": 0,
"closing-amount": 0,
"confirmed-round": 68203,
"fee": 1000,
"first-valid": 68200,
"genesis-hash": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"genesis-id": "mainnet-v1.0",
"id": "LNJH2Q2BYMV7KMMFMEEUCKIW6C3HPBOL4ARZBPPE2OBVVDPNVN6A",
"intra-round-offset": 0,
"last-valid": 69200,
"payment-transaction": {
"amount": 1000000000,
"close-amount": 0,
"receiver": "3P3CHL4M5JTDJKEL3ARLUZRXY23BWWPX6SZDC2NIBALL3SKE7JBIFOVOAY"
},
"receiver-rewards": 0,
"round-time": 1560638479,
"sender": "IVCEEIH2Q32DZNRTS5XFVEFFAQGERNZHHQT6S4UPY7ORJMHIQDSTX7YM4E",
"sender-rewards": 0,
"signature": {
"multisig": {
"subsignature": [
{
"public-key": "fDPiywtmtrpA2WOY+Mx9y6etNBCij1VKwZmGWW4PbKk="
},
{
"public-key": "RjQ91+zvYumrPm9UOEMN+GnlHW+0gliRCCV2b6KOlwk=",
"signature": "OZIfWu/UeOM4TqNTM2zZopJ3qoc6OHsHktVR4HSrWMUSWofIAtPgKf8Uqxcjkmu7E0k0O7spKtvqIEt53qTxAA=="
},
{
"public-key": "hYkIN+Iyt2675q+XuYwoAzwR8B0P17WTUFGYn456E4o=",
"signature": "kJ9eRUVplsmjPrtprSsZO/n5lG8Ff9YWhWKk3LQgtez9jGJ1EvQmxxh3nOWldYSkaDW+zUTGKHqDcsLleYqkAA=="
},
{
"public-key": "5ChQFEXiHWTeXoJCRymNn8rmEAJAxpaigu4wIgcaODU=",
"signature": "tjnxB0mpxOlTEbrJpdq888yyvuKRL2WJith8TXpVJO9fN8QnDCDAEsP2yGPiSlfQYQaqzKLe60DSIYll7nZeAg=="
},
{
"public-key": "ETnffVmxyVfJtVgCWFuStLsPJna9G1SHA1yJrfIo6RU="
},
{
"public-key": "k5F6WQJGyeiPHaN7fvmnBXz6YNq4NQ6BguE7yUmRWkI=",
"signature": "2RlNv5MxP528Zwh5lGSaY/vsCl50qwoxZ1/Blcu+mXLlG0Sr+3J44qRSjhbDrkwsJL5YHFcdv3uy3NSAd5s3AQ=="
}
],
"threshold": 4,
"version": 1
}
},
"tx-type": "pay"
},
{
"close-rewards": 0,
"closing-amount": 0,
"confirmed-round": 68208,
"fee": 1000,
"first-valid": 68000,
"genesis-hash": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
"genesis-id": "mainnet-v1.0",
"id": "3K7RW5MGZSUHMPHQT5MHV4UP6CWUKSI5UDXE5U7QVFHEB5GA3QEA",
"intra-round-offset": 0,
"last-valid": 69000,
"note": "MGY4MGUxYWI1OTVmYzE4MGM4ODIyMGMyMmRiMDNiMWIgOyA4ZWQ1ZTIwYWY0ZDhmNWZjMDY4ZjNkOGUyNzFkMzA1Yw==",
"payment-transaction": {
"amount": 100000,
"close-amount": 0,
"receiver": "ALGORANDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIN5DNAU"
},
"receiver-rewards": 0,
"round-time": 1560638501,
"sender": "3P3CHL4M5JTDJKEL3ARLUZRXY23BWWPX6SZDC2NIBALL3SKE7JBIFOVOAY",
"sender-rewards": 0,
"signature": {
"sig": "Keb0h37EURu7RxF/JKjeCbdsHKUtAE3mozDZ3Chx8xapvHJXhEz18+Xw/2X9eS0bTzawlQh8tu0u/90v4tjpAQ=="
},
"tx-type": "pay"
}
]
Loading