17 October, 2024
Reddit's upcoming changes attempt to safeguard the platform against AI crawlers
Reddit announced on Tuesday that it’s updating its Robots Exclusion Protocol (robots.txt file), which tells automated web bots whether they are permitted to crawl a site.
Historically, robots.txt file was used to allow search engines to scrape a site and then direct people to the content. However, with the rise of AI, websites are being scraped and used to train models without acknowledging the actual source of the content.
Along with the updated robots.txt file, Reddit will continue rate-limiting and blocking unknown bots and crawlers from accessing its platform. The company told TechCrunch that bots and crawlers will be rate-limited or blocked if they don’t abide by Reddit’s Public Content Policy and don’t have an agreement with the platform.
Reddit says the update shouldn’t affect the majority of users or good faith actors, like researchers and organizations, such as the Internet Archive. Instead, the update is designed to deter AI companies from training their large language models on Reddit content. Of course, AI crawlers could ignore Reddit’s robots.txt file.
The announcement comes a few days after a Wired investigation found that AI-powered search startup Perplexity has been stealing and scraping content. Wired found that Perplexity seems to ignore requests not to scrape its website, even though it blocked the startup in its robots.txt file. Perplexity CEO Aravind Srinivas responded to the claims and said that the robots.txt file is not a legal framework.
Reddit’s upcoming changes won’t affect companies that it has an agreement with. For instance, Reddit has a $60 million deal with Google that allows the search giant to train its AI models on the social platform’s content. With these changes, Reddit is signaling to other companies that want to use Reddit’s data for AI training that they will have to pay.
“Anyone accessing Reddit content must abide by our policies, including those in place to protect redditors,” Reddit said in its blog post. “We are selective about who we work with and trust with large-scale access to Reddit content.”
The announcement doesn’t come as a surprise, as Reddit released a new policy a few weeks ago that was designed to guide how Reddit’s data is being accessed and used by commercial entities and other partners.
Reddit's upcoming changes attempt to safeguard the platform against AI crawlers
Reddit announced on Tuesday that it’s updating its Robots Exclusion Protocol (robots.txt file), which tells automated web bots whether they are permitted to crawl a site.
Historically, robots.txt file was used to allow search engines to scrape a site and then direct people to the content. However, with the rise of AI, websites are being scraped and used to train models without acknowledging the actual source of the content.
Along with the updated robots.txt file, Reddit will continue rate-limiting and blocking unknown bots and crawlers from accessing its platform. The company told TechCrunch that bots and crawlers will be rate-limited or blocked if they don’t abide by Reddit’s Public Content Policy and don’t have an agreement with the platform.
Reddit says the update shouldn’t affect the majority of users or good faith actors, like researchers and organizations, such as the Internet Archive. Instead, the update is designed to deter AI companies from training their large language models on Reddit content. Of course, AI crawlers could ignore Reddit’s robots.txt file.
The announcement comes a few days after a Wired investigation found that AI-powered search startup Perplexity has been stealing and scraping content. Wired found that Perplexity seems to ignore requests not to scrape its website, even though it blocked the startup in its robots.txt file. Perplexity CEO Aravind Srinivas responded to the claims and said that the robots.txt file is not a legal framework.
Reddit’s upcoming changes won’t affect companies that it has an agreement with. For instance, Reddit has a $60 million deal with Google that allows the search giant to train its AI models on the social platform’s content. With these changes, Reddit is signaling to other companies that want to use Reddit’s data for AI training that they will have to pay.
“Anyone accessing Reddit content must abide by our policies, including those in place to protect redditors,” Reddit said in its blog post. “We are selective about who we work with and trust with large-scale access to Reddit content.”
The announcement doesn’t come as a surprise, as Reddit released a new policy a few weeks ago that was designed to guide how Reddit’s data is being accessed and used by commercial entities and other partners.
Ask Sophie: What changes are in store for PERM?
Sophie Alcorn
ContributorSophie Alcorn is the founder of Alcorn Immigration Law in Silicon Valley and 2019 Global Law Experts Awards’ “Law Firm of the Year in California for Entrepreneur Immigration Services.” She connects people with the businesses and opportunities that expand their lives.
More posts from Sophie Alcorn
Ask Sophie: What are the latest H-1B lottery updates?
Ask Sophie: What are the USCIS H-1B lottery and filing fee updates?
Sophie Alcorn, attorney, author and founder of Alcorn Immigration Law in Silicon Valley, California, is an award-winning Certified Specialist Attorney in Immigration and Nationality Law by the State Bar Board of Legal Specialization. Sophie is passionate about transcending borders, expanding opportunity, and connecting the world by practicing compassionate, visionary, and expert immigration law. Connect with Sophie on LinkedIn and Twitter.
TechCrunch+ members receive access to weekly “Ask Sophie” columns; use promo code ALCORN to purchase a one- or two-year subscription for 50% off.
Dear Sophie,
Our HR and operational consulting firm works primarily with tech startups. Would you provide an update on what we should look out for in the new year when it comes to the PERM process? Thanks!
— Hopeful HR
Dear Hopeful,
Happy New Year! I’m excited about what 2024 will bring in immigration policy changes designed to attract and retain international talent in STEM fields, particularly those spurred by President Biden’s executive order on AI.
If you haven’t already, talk with an immigration attorney about the complex PERM process, timing, risks and alternative options based on a company’s hiring situation and an employee’s immigration situation.
Now, let me provide a bit of context about where things currently stand with the PERM process before diving into the changes you should look out for that will — or will not 🙂 — impact PERM.
The current state of PERM
As you know, getting PERM labor certification from the U.S. Department of Labor (DOL) is the first step required for companies sponsoring current or prospective employees for an EB-2 advanced degree or exceptional ability green card or an EB-3 green card for professional workers. The PERM process aims to protect wages for Americans and establish that any qualified and available U.S. workers receive access to the job prior to offering a green card to the candidate.
If you’d like additional detail about the nuts and bolts of the PERM process, take a look at this previous Ask Sophie column.
In general, PERM requires employers to:
Determine the detailed duties and minimum requirements of the position, which will be used for the job posting during the recruitment process.File a prevailing wage request. The employer must pay the prevailing wage or higher based on the position and the location of the position to ensure that hiring a foreign national would not adversely affect the wages of U.S. workers.Go through an extensive recruitment process.Get labor certification from the Department of Labor.
Overall, the entire multistage labor certification process is currently taking between 18 and 24 months, as long as the DOL does not audit your case. According to the Foreign Labor Application Gateway (FLAG), the prevailing wage determination is taking about 8 to 12 months and the PERM itself is usually taking another 8 to 12 months.
The DOL conducts two types of audits: random audits and targeted audits. Random audits are done to make sure employers are following the PERM process. Some common reasons for targeted audits could include:
The employer laid off employees within the past six months. If the employer laid off workers within six months of filing the PERM application, they are required to notify the recently laid-off U.S. workers of the job opening and invite them to apply for the position.The candidate appears unqualified for the position.The role does not require a bachelor’s degree.A company executive is a relative of the candidate.
An audit does not mean an employer’s PERM will be denied, but it can add several months to the adjudication process. If an employer does not respond to the audit notice, the DOL will deem the case abandoned and the employer may be required to conduct supervised recruitment for any future PERM applications.
Once the DOL approves the PERM labor certification for that position, the employer needs to file the I-140 EB-2 or EB-3 green card petition with U.S. Citizenship and Immigration Services (USCIS) within 180 days.
Visa Bulletin outlook for EB-2 and EB-3
Given the slow-moving EB-2 and EB-3 green card lines, employers might want to consider supporting their employees to qualify for the EB-1 green card category, such as the EB-1A extraordinary ability green card or the EB-1C green card for multinational managers and executives.
Demand for EB-2 green cards from individuals born in countries other than India and China (referred to as the rest of the world) has been so high that for the first time ever, the category was not “current” at the beginning of the fiscal year 2024, which started on October 1, 2023. The same was true for EB-3, which was also not “current” at the start of the fiscal year for the first time since FY 2018.
The final action dates for the EB-2 and EB-3 categories for individuals born in all countries advanced in the January 2024 Visa Bulletin. Still, all EB-2 and EB-3 candidates will be waiting for a visa number to become available, and many observers predict that the EB-2 and EB-3 categories will not become current this fiscal year.
Bypassing PERM
Last month, the DOL published an RFI (request for information) to solicit public input on expanding its Schedule A shortage occupation list to include AI and other STEM-related occupations, as well as additional occupations for which the number of willing and qualified U.S. workers is insufficient to fill all positions. In this column, I discussed the Help Wanted Index, a data-driven method created by the Institute for Progress (IFP) to identify the occupations that should be on Schedule A.
Companies that employ a new or existing worker in an occupation listed in Schedule A can bypass the PERM labor certification process required for the EB-2 and EB-3 green cards.
The RFI is open for public input until 11:59 p.m. EST on February 20, 2024. I urge you to let the DOL know how crucial it is to add STEM-related occupations to Schedule A and to adopt the IFP’s data-driven method to identify those occupations. After the comment period ends, the DOL will review the comments and hopefully issue an updated Schedule A.
Prevailing wage rates
And there’s more good news for employers! Last year, the DOL postponed its plans to raise the prevailing wage rates of professional foreign workers that are used for both the PERM process for the EB-2 and EB-3 green cards and the Labor Condition Application (LCA) for the H-1B specialty occupation visas.
The DOL moved this proposal to raise the prevailing wage rates from its active rulemaking list to its long-term agenda, which experts say is usually the first step to the proposal being indefinitely shelved.
Clarity for international students
Last month, the USCIS issued policy guidance that offers clarity and reassurance to international students and any companies looking to sponsor them for an EB-2, EB-3, or other employment-based green card.
The F-1 and M-1 student visas are nonimmigrant visas, meaning F-1 and M-1 students must have nonimmigrant intent. In other words, they must demonstrate to immigration officials that they intend to remain in the U.S. temporarily and eventually plan to return to their home country.
Now, however, the USCIS has clarified “that F and M students must have a foreign residence that they do not intend to abandon, but that such students may be the beneficiary of a permanent labor certification application or immigrant visa petition and may still be able to demonstrate their intention to depart after a temporary period of stay.” That means students on F-1 and M-1 visas can be sponsored for employment-based green cards or self-petition for EB-1A or EB-2 NIW (National Interest Waiver) green cards without risking their current status.
Fewer EAD renewals
Last fall, the USCIS increased the validity of EAD (Employment Authorization Document) from two years to five years for individuals who have an approved green card petition and are waiting to adjust status, the final step in the green card process. That means individuals waiting for their priority date to become current to receive a green card won’t have to renew their EAD quite as often as before.
All in all, there’s been lots of good news for tech startups and their current employees and new hires they sponsor for EB-2 and EB-3 green cards.
Wishing you the best for this new year ahead!
— Sophie
Have a question for Sophie? Ask it here. We reserve the right to edit your submission for clarity and/or space.
The Sophie Alcorn Podcast follows origin stories of the heart. If you’d like to be a guest, she’s accepting applications!
Ask Sophie: What changes are in store for PERM?
Sophie Alcorn
Contributor
Sophie Alcorn is the founder of Alcorn Immigration Law in Silicon Valley and 2019 Global Law Experts Awards’ “Law Firm of the Year in California for Entrepreneur Immigration Services.” She connects people with the businesses and opportunities that expand their lives.
More posts from Sophie Alcorn
Ask Sophie: What are the latest H-1B lottery updates?
Ask Sophie: What are the USCIS H-1B lottery and filing fee updates?
Sophie Alcorn, attorney, author and founder of Alcorn Immigration Law in Silicon Valley, California, is an award-winning Certified Specialist Attorney in Immigration and Nationality Law by the State Bar Board of Legal Specialization. Sophie is passionate about transcending borders, expanding opportunity, and connecting the world by practicing compassionate, visionary, and expert immigration law. Connect with Sophie on LinkedIn and Twitter.
TechCrunch+ members receive access to weekly “Ask Sophie” columns; use promo code ALCORN to purchase a one- or two-year subscription for 50% off.
Dear Sophie,
Our HR and operational consulting firm works primarily with tech startups. Would you provide an update on what we should look out for in the new year when it comes to the PERM process? Thanks!
— Hopeful HR
Dear Hopeful,
Happy New Year! I’m excited about what 2024 will bring in immigration policy changes designed to attract and retain international talent in STEM fields, particularly those spurred by President Biden’s executive order on AI.
If you haven’t already, talk with an immigration attorney about the complex PERM process, timing, risks and alternative options based on a company’s hiring situation and an employee’s immigration situation.
Now, let me provide a bit of context about where things currently stand with the PERM process before diving into the changes you should look out for that will — or will not 🙂 — impact PERM.
The current state of PERM
As you know, getting PERM labor certification from the U.S. Department of Labor (DOL) is the first step required for companies sponsoring current or prospective employees for an EB-2 advanced degree or exceptional ability green card or an EB-3 green card for professional workers. The PERM process aims to protect wages for Americans and establish that any qualified and available U.S. workers receive access to the job prior to offering a green card to the candidate.
If you’d like additional detail about the nuts and bolts of the PERM process, take a look at this previous Ask Sophie column.
In general, PERM requires employers to:
Determine the detailed duties and minimum requirements of the position, which will be used for the job posting during the recruitment process.File a prevailing wage request. The employer must pay the prevailing wage or higher based on the position and the location of the position to ensure that hiring a foreign national would not adversely affect the wages of U.S. workers.Go through an extensive recruitment process.Get labor certification from the Department of Labor.
Overall, the entire multistage labor certification process is currently taking between 18 and 24 months, as long as the DOL does not audit your case. According to the Foreign Labor Application Gateway (FLAG), the prevailing wage determination is taking about 8 to 12 months and the PERM itself is usually taking another 8 to 12 months.
The DOL conducts two types of audits: random audits and targeted audits. Random audits are done to make sure employers are following the PERM process. Some common reasons for targeted audits could include:
The employer laid off employees within the past six months. If the employer laid off workers within six months of filing the PERM application, they are required to notify the recently laid-off U.S. workers of the job opening and invite them to apply for the position.The candidate appears unqualified for the position.The role does not require a bachelor’s degree.A company executive is a relative of the candidate.
An audit does not mean an employer’s PERM will be denied, but it can add several months to the adjudication process. If an employer does not respond to the audit notice, the DOL will deem the case abandoned and the employer may be required to conduct supervised recruitment for any future PERM applications.
Once the DOL approves the PERM labor certification for that position, the employer needs to file the I-140 EB-2 or EB-3 green card petition with U.S. Citizenship and Immigration Services (USCIS) within 180 days.
Visa Bulletin outlook for EB-2 and EB-3
Given the slow-moving EB-2 and EB-3 green card lines, employers might want to consider supporting their employees to qualify for the EB-1 green card category, such as the EB-1A extraordinary ability green card or the EB-1C green card for multinational managers and executives.
Demand for EB-2 green cards from individuals born in countries other than India and China (referred to as the rest of the world) has been so high that for the first time ever, the category was not “current” at the beginning of the fiscal year 2024, which started on October 1, 2023. The same was true for EB-3, which was also not “current” at the start of the fiscal year for the first time since FY 2018.
The final action dates for the EB-2 and EB-3 categories for individuals born in all countries advanced in the January 2024 Visa Bulletin. Still, all EB-2 and EB-3 candidates will be waiting for a visa number to become available, and many observers predict that the EB-2 and EB-3 categories will not become current this fiscal year.
Bypassing PERM
Last month, the DOL published an RFI (request for information) to solicit public input on expanding its Schedule A shortage occupation list to include AI and other STEM-related occupations, as well as additional occupations for which the number of willing and qualified U.S. workers is insufficient to fill all positions. In this column, I discussed the Help Wanted Index, a data-driven method created by the Institute for Progress (IFP) to identify the occupations that should be on Schedule A.
Companies that employ a new or existing worker in an occupation listed in Schedule A can bypass the PERM labor certification process required for the EB-2 and EB-3 green cards.
The RFI is open for public input until 11:59 p.m. EST on February 20, 2024. I urge you to let the DOL know how crucial it is to add STEM-related occupations to Schedule A and to adopt the IFP’s data-driven method to identify those occupations. After the comment period ends, the DOL will review the comments and hopefully issue an updated Schedule A.
Prevailing wage rates
And there’s more good news for employers! Last year, the DOL postponed its plans to raise the prevailing wage rates of professional foreign workers that are used for both the PERM process for the EB-2 and EB-3 green cards and the Labor Condition Application (LCA) for the H-1B specialty occupation visas.
The DOL moved this proposal to raise the prevailing wage rates from its active rulemaking list to its long-term agenda, which experts say is usually the first step to the proposal being indefinitely shelved.
Clarity for international students
Last month, the USCIS issued policy guidance that offers clarity and reassurance to international students and any companies looking to sponsor them for an EB-2, EB-3, or other employment-based green card.
The F-1 and M-1 student visas are nonimmigrant visas, meaning F-1 and M-1 students must have nonimmigrant intent. In other words, they must demonstrate to immigration officials that they intend to remain in the U.S. temporarily and eventually plan to return to their home country.
Now, however, the USCIS has clarified “that F and M students must have a foreign residence that they do not intend to abandon, but that such students may be the beneficiary of a permanent labor certification application or immigrant visa petition and may still be able to demonstrate their intention to depart after a temporary period of stay.” That means students on F-1 and M-1 visas can be sponsored for employment-based green cards or self-petition for EB-1A or EB-2 NIW (National Interest Waiver) green cards without risking their current status.
Fewer EAD renewals
Last fall, the USCIS increased the validity of EAD (Employment Authorization Document) from two years to five years for individuals who have an approved green card petition and are waiting to adjust status, the final step in the green card process. That means individuals waiting for their priority date to become current to receive a green card won’t have to renew their EAD quite as often as before.
All in all, there’s been lots of good news for tech startups and their current employees and new hires they sponsor for EB-2 and EB-3 green cards.
Wishing you the best for this new year ahead!
— Sophie
Have a question for Sophie? Ask it here. We reserve the right to edit your submission for clarity and/or space.
The Sophie Alcorn Podcast follows origin stories of the heart. If you’d like to be a guest, she’s accepting applications!
Google points to privacy changes, choice screens and data API ahead of DMA compliance day
Google has trailed another bundle of product tweaks ahead of Thursday’s deadline for compliance with the European Union’s Digital Markets Act (DMA).
The DMA applies to Alphabet, Google’s parent, which is one of six designated “gatekeepers” under the ex ante competition reform. The pan-EU regulation, which is intended to open up digital markets with measures to promote fairness and contestability, applies a set of upfront operational dos and don’ts across in-scope “core platform services.” Penalties for violating the regime can scale up to 10% of global annual turnover (or 20% for repeat offenders).
Incoming changes Google is flagging today ahead of the March 7 DMA compliance day include additional browser and search choice screens that will be shown to users of Android phones when they set up a device and (“soon”) to users of Chrome for desktop and iOS devices. The new choice screens will appear on or before March 6, per Google.
It says the design of the choice screens are “built on user research and testing, as well as feedback from the industry.” (Google’s eligibility criteria for browsers can be found here. Eligibility criteria for search engines is here.)
Today, Google is also announcing that it’s stopped the default-linking of personal data across user accounts for certain of its products, which it’s previously used to power “personalization” of content and ads. This pro-privacy change is happening because the DMA bans the use of people’s data for advertising without their consent. But Google isn’t going gently into this data goodnight.
“We currently share data across some Google products and services for certain purposes, including to help personalize your content and ads, depending on your settings. Today, users in the EEA [European Economic Area] can visit settings in their Google Account and choose if they want to continue to share data across Google services by linking them,” it writes, giving a backward take on the change that does its best to obfuscate the fact it’s being forced to stop this consentless tracking and profiling of its own users.
Google’s blog post further notes that users may see “new consent banners asking them whether they would like to link their Google services.” So the company appears to be intending to try to nudge users to re-enable its tracking — despite the DMA prohibiting the use of dark pattern designs that are intended to manipulate users into giving their consent.
The adtech giant is the DMA gatekeeper with by far the largest number of regulated platforms — eight in total; namely Google Maps, Google Play, Google Shopping, Google Ads, Chrome, Android, Google Search and YouTube. Which is why it’s setting out such a smorgasbord of changes across multiple products.
Elsewhere on data for ads, Google appears to be relying on its advertisers to keep a pipeline of targetable user data flowing through its ad engines. Its blog post highlights that it’s making “multiple upgrades to our advertising products and tools to help advertisers communicate consent for data they collect” in accordance with what it describes as its “long standing EU end user consent policy.“
Google offers no explanation of the nature of the changes here. And it will be for regulators to scrutinize its outsourcing of consent for tracking to third parties. (NB: Compliance with the EU’s data protection framework, the GDPR, is also required of DMA gatekeepers — and on that front it’s worth noting there’s a long-standing GDPR complaint against Google’s adtech sitting with the Irish Data Protection Commission.)
The DMA also requires Google to provide advertiser customers with more information about their ads. On this, Google notes advertisers and publishers in the EEA will “be able to receive some additional data” — without its blog post specifying exactly what they’ll get. It just stipulates the data will be “shared in a way that protects user privacy and customers’ commercially-sensitive information.”
In another DMA-driven change that’s slated to start on March 6, Google says it will launch a program allowing Android developers of Play-distributed apps to “directly lead users in the EEA outside the app, including to promote offers.” This suggests Google will let these developers include links in their software to direct users to cheaper deals elsewhere — something it has previously not allowed them to do under anti-steering restrictions, unless they eschewed distribution in its popular app store.
The change should make it easier for Android developers to direct users to their own websites to take payments for subscriptions etc. to avoid reduce(?)* the commission fee Google charges through its app store — potentially boosting their profitability. Google got in touch to say it plans to take a cut of these sales. But it has not yet confirmed what its take will be. “Fees will continue to apply,” a spokesperson told TechCrunch. “We’ll be sharing more details this week on this.”
Google has also announced the impending launch (“this week”) of a data portability API for developers in the EEA — to meet new requirements in the DMA.
Yesterday TikTok, which is also in scope of the DMA, announced its own API for data portability. All the gatekeepers will have to do this.
EU lawmakers hope the regulation’s data portability requirements will fire up competition against gatekeepers by facilitating service switching and/or multi-homing — making it easier for users to port their data to a third-party app or service and for businesses to get access to data so they can cater to users.
Despite a high level of DMA risk for Google, on account of how many of its services are in-scope of the regulation, at times its blog post spins the idea it’s already well on the way to being compliant with the regulation as it suggests it already fulfills some of the requirements — such as the ability for users of its mobile platform Android to install alternative app stores and sideload apps.
Some of what Google is trailing in today’s blog post also appears to be re-announcing changes previously announced and/or launched in recent weeks. Such as tweaks to regional search results that removed its flight aggregator unit but added some new box-outs and buttons. Those tweaks quickly garnered a furious reaction from comparison sites — which accuse Google of devising and launching a new service (in the form of rich content features) that breaches the regulation’s ban on self-preferencing.
On March 7, the enforcement era of the DMA will kick off — with gatekeepers’ compliance reports being made public and the European Commission inviting stakeholders to a series of workshops to give detailed initial feedback.
The Commission is the sole enforcer of the DMA. The bloc can instigate investigations where it suspects noncompliance by gatekeepers. It also has powers to deploy interim measures — so has the tools to move quickly on pressing issues. It’s also fair to say the EU is under considerable pressure — including from a reputational perspective — to deliver strong enforcement of this flagship digital reform so the Commission will need to hit the ground running.
*This report was updated after Google told us the incoming changes to its anti-steering measures for developers distributing apps via its Play Store, which will let them direct users to external offers, will not let them avoid its fee, as our report originally suggested. Google told us “fees will continue to apply.” It did not confirm what the fees will be but said it will be revealing details later this week
Google’s search tweaks draw fire as EU self-preferencing ban looms
TikTok launches data portability API ahead of Europe’s DMA regulatory deadline