Web scraping

[tavster]

Any DDFer with good knowledge of 1) web scraping and of 2) the different cloud-based hosting solutions out there?

I have a few questions to PM

Thanks much

[Alexsei]

Fiverr's are fantastic for these stuff

[tavster]

Fiverr's are fantastic for these stuff

Thanks. Tried that. So far I only came across people which would run the requests for me on a ad-hoc basis.

Im looking for build the a sustainable system for me. I PMd a few people on Fiverr.. let's see

[Ver hut gazugt]

Thanks. Tried that. So far I only came across people which would run the requests for me on a ad-hoc basis.

Im looking for build the a sustainable system for me. I PMd a few people on Fiverr.. let's see

A lot depends on what you are trying to scrape. There are services out there that do it for on a system style bases (api) not very costly compared to keep on fixing your own system.

[tavster]

A lot depends on what you are trying to scrape. There are services out there that do it for on a system style bases (api) not very costly compared to keep on fixing your own system.

Thanks.

I'm looking to build something basic and bespoke as there is no existing set API for it. Any good programer can probably do it in an hour.

[Euclid]

Any good programmer can probably do it in an hour.

Famous last words

If only I'd had a nickel for each time I've heard that

[tavster]

Famous last words

If only I'd had a nickel for each time I've heard that

+1
The fact that they charge 10 hours for something they do in one hour is a different story 

[Euclid]

+1
The fact that they charge 10 hours for something they do in one hour is a different story 

You misunderstood me. I'm in software and I don't take freelance jobs for this reason; the expectation  of work needed doesn't match the actual work that's required. "Just", "only", "simple" are all words which are bandied about when it comes to someone looking for a tech solution, when they have 1) no idea what they really want and 2) no idea the amount of work and knowledge which is actually necessary.

/rant


(Sorry to take it off topic; but if you don't get any hits on your request, it's possible because other software guys feel the same as I do. Though web scraping is a solved problem so you'll likely be able to use an out of the box solution such as Scrapy.)

[tavster]

You misunderstood me. I'm in software and I don't take freelance jobs for this reason; the expectation  of work needed doesn't match the actual work that's required. "Just", "only", "simple" are all words which are bandied about when it comes to someone looking for a tech solution, when they have 1) no idea what they really want and 2) no idea the amount of work and knowledge which is actually necessary.

/rant

I indeed misunderstood you.
Meant no disrespect. I would have no issue with a free lancer telling me it would take him 5 hours and not one, if he runs me through the process.

Regarding the solutions out there, i feel that there are either very basic no-code ones which don't do what I need.

Or more advanced one which require some coding knowledge.

I might just go with the latter for the fun of it

[Euclid]

I indeed misunderstood you.
Meant no disrespect. I would have no issue with a free lancer telling me it would take him 5 hours and not one, if he runs me through the process.

Regarding the solutions out there, i feel that there are either very basic no-code ones which don't do what I need.

Or more advanced one which require some coding knowledge.

I might just go with the latter for the fun of it

If you're somewhat familiar with Python I'd recommend that you try https://scrapy.org

You can post any questions you have here and I'll try to help.

[Ver hut gazugt]

I indeed misunderstood you.
Meant no disrespect. I would have no issue with a free lancer telling me it would take him 5 hours and not one, if he runs me through the process.

Regarding the solutions out there, i feel that there are either very basic no-code ones which don't do what I need.

Or more advanced one which require some coding knowledge.

I might just go with the latter for the fun of it

I am not a programmer but from my experience  almost nothing takes only five hours. Hello world takes five hours and if your needs do not come in an out of the box solution it’s guaranteed to take wayyyyy longer 

[etech0]

Thanks.

I'm looking to build something basic and bespoke as there is no existing set API for it. Any good programer can probably do it in an hour.

Have you tried google sheets and IMPORTXML? If it'll work for you, you can likely do it yourself.

[ushdadude]

Never heard of web scraping before but it's n interesting idea. Would it work on a password protected site?
I want to switch EHRs and they are notoriously difficult in allowing you to export your records. Would something like this be a viable solution?

[Yo ssi]

Never heard of web scraping before but it's n interesting idea. Would it work on a password protected site?
I want to switch EHRs and they are notoriously difficult in allowing you to export your records. Would something like this be a viable solution?

While anything is possible, it doesn't seem like the proper use.

[yesitsme]

You misunderstood me. I'm in software and I don't take freelance jobs for this reason; the expectation  of work needed doesn't match the actual work that's required. "Just", "only", "simple" are all words which are bandied about when it comes to someone looking for a tech solution, when they have 1) no idea what they really want and 2) no idea the amount of work and knowledge which is actually necessary.

/rant


(Sorry to take it off topic; but if you don't get any hits on your request, it's possible because other software guys feel the same as I do. Though web scraping is a solved problem so you'll likely be able to use an out of the box solution such as Scrapy.)

I love the quick questions

[Sammy82]

Never heard of web scraping before but it's n interesting idea. Would it work on a password protected site?
I want to switch EHRs and they are notoriously difficult in allowing you to export your records. Would something like this be a viable solution?

Are you sure that there's nothing about that in the contract? I can't imagine that there isn't something about who owns the data....

[yelped]

Are you sure that there's nothing about that in the contract? I can't imagine that there isn't something about who owns the data....

EHR? There's no way the third party is allowed to own the PHI legally (cmiiw).

[Sammy82]

Our policy is the data is ours as long as we service you once you leave its all your

Doesn't sound right. Don't what what line of healthcare you're in but usually the provider has to keep records for a number of years (6-10+ here in NY). Unless you only accept private pay. In which case maybe your hiring? :-)
I'm not a pro but definitely look at the actual contract and/or speak to a competent attorney.

[ushdadude]

Are you sure that there's nothing about that in the contract? I can't imagine that there isn't something about who owns the data....

Good point. I'll check

[AsherO]

Never heard of web scraping before but it's n interesting idea. Would it work on a password protected site?
I want to switch EHRs and they are notoriously difficult in allowing you to export your records. Would something like this be a viable solution?

EHRs are complicated datasets. The vendor you want to move to can’t help you to migrate your data?