I miss nodejs 
why is python like this
1 Like
that was it, thank you guys!
i notice that currently the article URL does not get included. Would it be possible to include https://fc2ppvdb.com/articles/ AND https://adult.contents.fc2.com/article/ URLs since that is how i keep track which scraper i used on scenes since fc2ppvdb does not have cover images but fc2 does.
leaning no on adding fc2.com URLs since there’s already another scraper for that. [fc2ppvdb] add url to result · stashapp/CommunityScrapers@c5ccf43 · GitHub
fair enough. im struggling a bit adding it just for myself how would i add it to the python script since it already has a url value?
you are right. its probably best to just first use the FC2 scraper for cover images, filter for them to use the fc2ppvdb scraper for the actors, then the scene has both urls
apparently the site is switching to https://fc2cmadb.com, i hope most of the work can be carried over. at least this one has images (so far).
I modified the script to work on the new domain. @feederbox826 could you please review the code and update it in the repository. Pastebin is up for a week from now, but I’ll reupload if necessary.
Maybe the script could use a rename to reflect the new domain, but otherwise it should be functionally identical to the previous one. There’s no more need for ‘age_pass’ in the configuration.
Edit: the new API seems to return 404 for some ID’s that are functioning when using browser. I haven’t tested that many ID’s yet, but my guess is around 1 in 20 fails. That may depend on how old or new the ID’s are or something like that. I haven’t been able to figure out the cause for this or anything common to the ID’s that fail. Any help with this is appreciated.
1 Like
thanks! Idk if it my setup but it gets stuck on [Scrape / fc2cmadb] getting fresh cloudflare cookies for me even on different scenes and IPs
That part of the code has not changed. While testing I also got a period when I couldn’t get CF cookies for whatever reason. I didn’t investigate it more because it started working again after few hours. Hopefully that’s the case for you too. If not you should check flaresolverr logs for any hints.
Regarding the 404’s that the API returns. There’s much more of them than 1 in 20 I’m afraid. I have also managed to get a valid scrape of an ID that previously returned 404 so I guess the situation is dependent on the state of the ID in the backend and there’s not much we can do about it. Other than scraping the HTML which doesn’t sound like a great solution.
1 Like
you are right. it works now thank you very much.
Since this site has images i added the following to your script
scene: ScrapedScene = {}
scene["image"] = article.get("image_url", "")
Link doesn’t work, says it was removed from pastebin
Yeah, sorry the link expired. Here’s the whole thing including @fc2_lover image_url addition. This one will be up six months from now, hopefully we can get the code to the repository by then.
import json
import re
import sys
import os
from py_common import log
from py_common.types import ScrapedScene
from py_common.util import scraper_args, dig
from py_common.config import get_config
ensure_deps = ["requests"]
import requests
FLARESOLVERR_URL = os.environ.get("FLARESOLVERR_URL", "http://localhost:8191/v1")
# fc2cmadb_session has to be manually defined and passed in
# but is **extremely** long lived (token from 5mos ago still valid????)
config = get_config(
default="""
# F12 -> Application -> Cookies
fc2cmadb_session =
xsrf_token =
"""
)
def check_login(text):
if 'https://fc2cmadb.com/login' in text.lower():
log.error("Login prompt detected. Check cookies and try again.")
log.error("fc2ppv_session ends with %3D")
log.debug(f"fc2cmadb_session cookie: {config['fc2cmadb_session']}")
log.debug(f"XSRF-TOKEN cookie: {config['xsrf_token']}")
return True
return False
def get_flaresolverr_soln(url):
response = requests.post(FLARESOLVERR_URL, json={
"cmd": "request.get",
"url": url,
"cookies": [
{ "name": "fc2cmadb-session", "value": config["fc2cmadb_session"] },
{ "name": "XSRF-TOKEN", "value": config["xsrf_token"] },
],
"session_ttl_minutes": 5, # destroy session after 5 minutes
})
if response.status_code != 200:
raise Exception(f"FlareSolverr request failed with status code {response.status_code}: {response.text}")
return response.json().get("solution")
def scene_from_url(url: str) -> ScrapedScene:
# if no config, throw error
if not config["fc2cmadb_session"] or not config["xsrf_token"]:
log.error("Missing required cookies in config. Please update config and try again.")
log.debug(f"fc2cmadb_session cookie: {config['fc2cmadb_session']}")
log.debug(f"XSRF-TOKEN cookie: {config['xsrf_token']}")
return {}
log.debug("getting fresh cloudflare cookies")
# get solution
solution = get_flaresolverr_soln(url)
# set cookies from solution
session = requests.Session()
# disable proxies (ASN blocked)
session.proxies = {}
session.trust_env = False
# end disable proxies
soln_cookies = solution.get("cookies", [])
for cookie in soln_cookies:
session.cookies.set(cookie['name'], cookie['value'], domain=cookie['domain'], path=cookie['path'])
session.headers.update({"User-Agent": solution.get("userAgent")})
# add get solution cookies
session.cookies.set("ageVerified", "true", domain="fc2cmadb.com", path="/")
session.cookies.set("fc2cmadb-session", config["fc2cmadb_session"], domain=".fc2cmadb.com", path="/")
session.cookies.set("XSRF-TOKEN", config["xsrf_token"], domain=".fc2cmadb.com", path="/")
init = session.get(url)
# check for CF ASN block
if init.status_code == 403 and "1005" in init.text:
log.error("Cloudflare ASN block detected. This scraper will not work from this IP address.")
return {}
if check_login(init.text):
return {}
# we need the inertia version from the app script
pattern = r'<script[^>]*data-page="app"[^>]*>(.*?)</script>'
match = re.search(pattern, init.text, re.DOTALL)
if not match:
log.error("Can't extract inertia version.")
return {}
json_content = match.group(1)
data = json.loads(json_content)
inertia_version = data.get('version')
log.debug(f"cookies set, hitting article info endpoint {url}")
# fill inertia headers to get json instead of html
session.headers.update({"X-Inertia": "true"})
session.headers.update({"X-Requested-With": "XMLHttpRequest"})
session.headers.update({"X-Inertia-Partial-Component": "Articles/Show"})
session.headers.update({"X-Inertia-Partial-Data": "article,actresses"})
session.headers.update({"X-Inertia-Version": inertia_version})
session.headers.update({"Referer": url})
session.headers.update({"Accept": "text/html, application/xhtml+xml"})
session.headers.update({"Cache-Control": "no-cache"})
try:
info_res = session.get(url)
if check_login(info_res.text):
return {}
resp_json = info_res.json()
except json.JSONDecodeError:
log.error("Invalid JSON response from article info endpoint. Try again.")
return {}
article = resp_json.get("props", {}).get('article')
actresses = resp_json.get("props", {}).get('actresses', [])
if not article:
log.error(f"Can't get article data from {url}")
log.debug(info_res)
scene: ScrapedScene = {}
# images - all removed, even the uncensored ones
scene["title"] = article.get("title", "").strip()
scene["code"] = "FC2-PPV-" + str(article.get("video_id", ""))
scene["date"] = article.get("release_date", "")
scene["studio"] = { "name": dig(article, "writer", "name") }
scene["tags"] = [{ "name": tag.get("name", "").strip() } for tag in article.get("tags", [])]
scene["performers"] = [{ "name": performer.get("name", "").strip() } for performer in actresses]
scene["urls"] = [url]
scene["image"] = article.get("image_url", "")
return scene
def url_from_frag(files) -> str:
filename = files[0].get("path") if files else None
basename = os.path.basename(filename) if filename else None
if not basename:
return ""
code = re.search(r"(\d{5,})", basename)
match = code.group(1) if code else None
if match:
return f"https://fc2cmadb.com/articles/{match}"
return ""
if __name__ == "__main__":
op, args = scraper_args()
match op, args:
case "scene-by-url", {"url": url} if url:
result = scene_from_url(url)
case "scene-by-fragment" | "scene-by-query-fragment", args:
files = args.get("files", [])
url = url_from_frag(files)
if url:
result = scene_from_url(url)
else:
log.error("Could not extract article ID from filename")
sys.exit(1)
case _:
log.error(f"Operation: {op}, arguments: {json.dumps(args)}")
sys.exit(1)
print(json.dumps(result))
2 Likes
it would be great if the unique actresses ID could be passed along the performer name in disambiguation to prevent name collision since a lot of them have the same name. actress_id does not seem to work sadly.
I have been patiently adding ID as disambiguation by hand when needed 
Took now time to add it to the script. I think there’s enough same names to disambiguate all names from fc2cmadb (= I tried to think a way to disambiguate only names with duplicates but failed).
Before starting to scrape scenes with the updated script it’s best to update the current performers so you start getting correct performer matches on the new scene scapes.
For this I added performerByURL functionality to script. For each performer you want to update you’ll need to go and edit that performer, add valid fc2cmadb actress url to the URLs field (https://fc2cmadb.com/actresses/[ID]). After that the “Scrape” button next to the url becomes active and you can scrape the performer, which adds the disambiguation field (fc2cmadb-[ID]).
Edit: the actress scrape doesn’t add much value. The alias field seem to occasionally contain tags as well so I don’t think we can use it. There are no actress pictures to use. So if you’re mass editing existing performers it might be faster just manually adding the “fc2cmadb-[ID]” disambiguation by hand.
Edit2: cleaned up the script a bit. The intermittent 404 I was getting before turned out to be expiring login cookies. It didn’t affect all IDs so I thought there was some other problem, but copying fresh cookies has helped at least for now.
Updated fc2ppvdb.yml
name: fc2ppvdb
sceneByURL:
- action: script
url:
- fc2cmadb.com/articles/
script:
- python
- fc2ppvdb.py
- scene-by-url
sceneByFragment:
action: script
script:
- python
- fc2ppvdb.py
- scene-by-fragment
performerByURL:
- action: script
url:
- fc2cmadb.com/actresses/
script:
- python
- fc2ppvdb.py
- performer-by-url
Updated fc2ppvdb.py
import json
import os
import re
import requests
import sys
from py_common import log
from py_common.types import ScrapedScene, ScrapedPerformer
from py_common.util import scraper_args, dig
from py_common.config import get_config
from requests import Session
FLARESOLVERR_URL = os.environ.get("FLARESOLVERR_URL", "http://localhost:8191/v1")
# fc2cmadb_session has to be manually defined and passed in
# but is **extremely** long lived (token from 5mos ago still valid????)
config = get_config(
default="""
# F12 -> Application -> Cookies
fc2cmadb_session =
xsrf_token =
"""
)
def check_login(text):
if 'https://fc2cmadb.com/login' in text.lower():
log.error("Login prompt detected. Check cookies and try again.")
log.error("fc2ppv_session ends with %3D")
log.debug(f"fc2cmadb_session cookie: {config['fc2cmadb_session']}")
log.debug(f"XSRF-TOKEN cookie: {config['xsrf_token']}")
return True
return False
def get_flaresolverr_soln(url):
response = requests.post(FLARESOLVERR_URL, json={
"cmd": "request.get",
"url": url,
"cookies": [
{ "name": "fc2cmadb-session", "value": config["fc2cmadb_session"] },
{ "name": "XSRF-TOKEN", "value": config["xsrf_token"] },
],
"session_ttl_minutes": 5, # destroy session after 5 minutes
})
if response.status_code != 200:
raise Exception(f"FlareSolverr request failed with status code {response.status_code}: {response.text}")
return response.json().get("solution")
def get_valid_image_url(url: str) -> str:
return url if (url and url.startswith(('http://', 'https://')) and not url.endswith("no-image.jpg")) else ""
def get_session(url: str) -> Session:
# if no config, throw error
if not config["fc2cmadb_session"] or not config["xsrf_token"]:
log.error("Missing required cookies in config. Please update config and try again.")
log.debug(f"fc2cmadb_session cookie: {config['fc2cmadb_session']}")
log.debug(f"XSRF-TOKEN cookie: {config['xsrf_token']}")
print("null")
exit(1)
log.debug("getting fresh cloudflare cookies")
# get solution
solution = get_flaresolverr_soln(url)
# set cookies from solution
session = requests.Session()
# disable proxies (ASN blocked)
session.proxies = {}
session.trust_env = False
# end disable proxies
soln_cookies = solution.get("cookies", [])
for cookie in soln_cookies:
session.cookies.set(cookie['name'], cookie['value'], domain=cookie['domain'], path=cookie['path'])
session.headers.update({"User-Agent": solution.get("userAgent")})
# add get solution cookies
session.cookies.set("ageVerified", "true", domain="fc2cmadb.com", path="/")
session.cookies.set("fc2cmadb-session", config["fc2cmadb_session"], domain=".fc2cmadb.com", path="/")
session.cookies.set("XSRF-TOKEN", config["xsrf_token"], domain=".fc2cmadb.com", path="/")
init = session.get(url)
# check for CF ASN block
if init.status_code == 403 and "1005" in init.text:
log.error("Cloudflare ASN block detected. This scraper will not work from this IP address.")
print("null")
exit(1)
if check_login(init.text):
print("null")
exit(1)
# we need the inertia version from the app script
pattern = r'<script[^>]*data-page="app"[^>]*>(.*?)</script>'
match = re.search(pattern, init.text, re.DOTALL)
if not match:
log.error("Can't extract inertia version.")
print("null")
exit(1)
json_content = match.group(1)
data = json.loads(json_content)
inertia_version = data.get('version')
# fill inertia headers to get json instead of html
session.headers.update({"X-Inertia": "true"})
session.headers.update({"X-Requested-With": "XMLHttpRequest"})
session.headers.update({"X-Inertia-Version": inertia_version})
session.headers.update({"Referer": url})
session.headers.update({"Accept": "text/html, application/xhtml+xml"})
session.headers.update({"Cache-Control": "no-cache"})
return session
def get_payload_from_url(session: Session, url: str):
try:
info_res = session.get(url)
if check_login(info_res.text):
print("null")
exit(1)
resp_json = info_res.json()
except json.JSONDecodeError:
log.error("Invalid JSON response from article info endpoint. Try again.")
print("null")
exit(1)
if resp_json.get("component", "") == "Error":
log.error("API returned error with status: " + str(resp_json.get("props").get("status")))
print("null")
exit(1)
return resp_json
def scene_from_url(url: str) -> ScrapedScene:
session = get_session(url)
session.headers.update({"X-Inertia-Partial-Component": "Articles/Show"})
session.headers.update({"X-Inertia-Partial-Data": "article,actresses"})
log.debug(f"cookies set, hitting article info endpoint {url}")
resp_json = get_payload_from_url(session, url)
article = resp_json.get("props").get('article')
actresses = resp_json.get("props").get('actresses', [])
if not article:
log.error(f"Can't get article data from {url}")
log.debug(resp_json)
print("null")
exit(1)
scene: ScrapedScene = {}
scene["title"] = article.get("title", "").strip()
scene["code"] = "FC2-PPV-" + str(article.get("video_id", ""))
scene["date"] = article.get("release_date", "")
scene["studio"] = { "name": dig(article, "writer", "name") }
scene["tags"] = [{ "name": tag.get("name", "").strip() } for tag in article.get("tags", [])]
scene["urls"] = [url]
scene["image"] = get_valid_image_url(article.get("image_url", ""))
scene["performers"] = []
for actress in actresses:
scene["performers"].append(get_performer_from_actress(actress))
return scene
def get_performer_from_actress(actress) -> ScrapedPerformer:
performer: ScrapedPerformer = {
"name": actress.get("name", "").strip(),
"disambiguation": "fc2cmadb-" + str(actress.get("id")),
"urls": [ "https://fc2cmadb.com/actresses/" + str(actress.get("id")) ],
"gender": "female",
"image": get_valid_image_url(actress.get("image_url", ""))
}
return performer
def performer_from_url(url: str) -> ScrapedPerformer:
session = get_session(url)
session.headers.update({"X-Inertia-Partial-Component": "Actresses/Show"})
session.headers.update({"X-Inertia-Partial-Data": "actress"})
log.debug(f"cookies set, hitting actress info endpoint {url}")
resp_json = get_payload_from_url(session, url)
actress = resp_json.get("props").get('actress')
if not actress:
log.error(f"Can't get actress data from {url}")
log.debug(resp_json)
print("null")
exit(1)
return get_performer_from_actress(actress)
def url_from_frag(files) -> str:
filename = files[0].get("path") if files else None
basename = os.path.basename(filename) if filename else None
if not basename:
return ""
code = re.search(r"(\d{5,})", basename)
match = code.group(1) if code else None
if match:
return f"https://fc2cmadb.com/articles/{match}"
return ""
if __name__ == "__main__":
op, args = scraper_args()
match op, args:
case "scene-by-url", {"url": url} if url:
result = scene_from_url(url)
case "scene-by-fragment" | "scene-by-query-fragment", args:
files = args.get("files", [])
url = url_from_frag(files)
if url:
result = scene_from_url(url)
else:
log.error("Could not extract article ID from filename")
print("null")
exit(1)
case "performer-by-url", {"url": url} if url:
result = performer_from_url(url)
case _:
log.error(f"Operation: {op}, arguments: {json.dumps(args)}")
print("null")
exit(1)
print(json.dumps(result))