SubCrawler
节点爬取,筛选, google ping,支持Clash,base64订阅解析,生成可用的ss, ssr, v2ray, trojan,vless节点. (若不进行二次开发,请不要fork)
Top Related Projects
Utility to convert between various subscription format
SSR 去广告ACL规则/SS完整GFWList规则/Clash规则碎片,Telegram频道订阅地址
机场推荐/SSR V2ray节点订阅机场/镜像直连/工具推荐
Quick Overview
SubCrawler is a tool for collecting and aggregating subscription links from various sources. It's designed to help users gather and manage subscription information for services like VPN or proxy servers. The project is primarily written in Kotlin and includes features for link parsing, validation, and output generation.
Pros
- Supports multiple subscription link formats and sources
- Includes built-in link validation and deduplication
- Offers various output formats for collected links
- Regularly updated with new features and improvements
Cons
- Limited documentation, which may make it challenging for new users
- Primarily focused on Chinese language sources and services
- May require additional configuration for use with specific services
- Potential legal and ethical concerns regarding the collection and sharing of subscription links
Code Examples
// Example 1: Parsing a subscription link
val parser = SubscriptionParser()
val subscriptionInfo = parser.parse("https://example.com/subscription")
println(subscriptionInfo)
// Example 2: Validating a list of links
val validator = LinkValidator()
val validLinks = validator.validateLinks(listOfLinks)
validLinks.forEach { println(it) }
// Example 3: Generating output in a specific format
val generator = OutputGenerator()
val jsonOutput = generator.generateJson(validLinks)
println(jsonOutput)
Getting Started
To use SubCrawler in your project:
-
Clone the repository:
git clone https://github.com/Leon406/SubCrawler.git
-
Build the project using Gradle:
./gradlew build
-
Run the main application:
java -jar build/libs/SubCrawler.jar
-
Configure the sources and output options in the
config.properties
file. -
Use the generated output files or integrate the library into your own Kotlin/Java project.
Competitor Comparisons
Utility to convert between various subscription format
Pros of subconverter
- More comprehensive feature set for subscription conversion and management
- Supports a wider range of proxy protocols and configuration formats
- Better documentation and user guides
Cons of subconverter
- More complex setup and configuration process
- Heavier resource usage due to its extensive features
- Steeper learning curve for new users
Code Comparison
SubCrawler (Kotlin):
fun parseClash(text: String): List<Sub> {
return parseYaml(text).proxies?.map { proxy ->
Sub(
type = proxy.type,
name = proxy.name,
server = proxy.server,
port = proxy.port
)
} ?: emptyList()
}
subconverter (C++):
void rulesetToClash(YAML::Node &base_rule, std::string &rule_name, std::string &custom_port)
{
base_rule["name"] = rule_name;
base_rule["type"] = "http";
base_rule["behavior"] = "classical";
base_rule["path"] = "./ruleset/" + rule_name + ".yaml";
if(!custom_port.empty())
base_rule["url"] = "http://www.gstatic.com/generate_204";
}
The code snippets show different approaches to handling configuration data, with SubCrawler focusing on parsing and subconverter on generating rule sets.
SSR 去广告ACL规则/SS完整GFWList规则/Clash规则碎片,Telegram频道订阅地址
Pros of ACL4SSR
- More comprehensive rule sets for various applications and services
- Regular updates and maintenance by an active community
- Includes pre-configured lists for popular platforms like Clash and Quantumult X
Cons of ACL4SSR
- Primarily focused on Chinese websites and services
- May require more manual configuration for non-Chinese users
- Less flexibility in customizing individual rules compared to SubCrawler
Code Comparison
SubCrawler:
fun crawlSubscription(url: String): List<String> {
val response = client.newCall(Request.Builder().url(url).build()).execute()
return response.body?.string()?.split("\n") ?: emptyList()
}
ACL4SSR:
payload:
- DOMAIN-SUFFIX,googleapis.cn
- DOMAIN-SUFFIX,googleapis.com
- DOMAIN-SUFFIX,google.cn
- DOMAIN-SUFFIX,google.com
SubCrawler focuses on crawling and processing subscription URLs, while ACL4SSR provides pre-defined rule sets in YAML format. SubCrawler offers more flexibility for custom crawling and processing, whereas ACL4SSR provides ready-to-use rules for popular services and platforms.
Pros of sub-web
- User-friendly web interface for subscription conversion and management
- Supports multiple subscription formats and protocols
- Easy deployment with Docker
Cons of sub-web
- Limited crawling capabilities compared to SubCrawler
- Less frequent updates and maintenance
- Fewer advanced features for proxy testing and filtering
Code Comparison
sub-web (Vue.js component):
<template>
<div class="subscription-card">
<el-input v-model="url" placeholder="Enter subscription URL"></el-input>
<el-button @click="convertSubscription">Convert</el-button>
</div>
</template>
SubCrawler (Kotlin code):
fun crawlSubscriptions(urls: List<String>): List<Proxy> {
return urls.flatMap { url ->
val response = httpClient.get(url)
parseProxies(response.body)
}
}
Summary
SubCrawler focuses on crawling and aggregating proxy subscriptions, offering more advanced features for proxy testing and filtering. It's primarily a command-line tool written in Kotlin.
sub-web provides a web-based interface for subscription conversion and management, making it more accessible for users who prefer a graphical interface. It's built with Vue.js and supports various subscription formats.
Both projects serve different use cases within the proxy management ecosystem, with SubCrawler being more suitable for advanced users and automation, while sub-web caters to those seeking a user-friendly web interface for subscription handling.
机场推荐/SSR V2ray节点订阅机场/镜像直连/工具推荐
Pros of Share-SSR-V2ray
- Provides a curated list of SSR and V2Ray servers, making it easier for users to find reliable connections
- Includes detailed instructions and tutorials for setting up various VPN clients
- Regularly updated with new server information and configuration guides
Cons of Share-SSR-V2ray
- Less automated than SubCrawler, requiring manual updates and maintenance
- Limited to specific VPN protocols (SSR and V2Ray), while SubCrawler supports a wider range
- Lacks built-in tools for testing or validating server connections
Code Comparison
Share-SSR-V2ray (README.md):
## SSR 订阅链接
- https://raw.githubusercontent.com/ssrsub/ssr/master/ssrsub
- https://www.liesauer.net/yogurt/subscribe?ACCESS_TOKEN=DAYxR3mMaZAsaqUb
SubCrawler (build.gradle.kts):
dependencies {
implementation("org.jsoup:jsoup:1.14.3")
implementation("com.squareup.okhttp3:okhttp:4.9.3")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.0")
}
The code comparison shows that Share-SSR-V2ray primarily consists of markdown files with server information, while SubCrawler is a Kotlin-based project with dependencies for web scraping and asynchronous programming.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
è®¿å®¢æ° :eyes:
Telegram群ç»
ä¼è´¨èç¹æ±
大é¨åé½åç¨,èªè¡æç´¢
èç¹æ± æ建
ææå¡å¨çå¯èªè¡æ建 proxypool é ç½®æ件 source.yaml
订é 转æ¢
èç¹æµé
-
æ¬å°æµé
-
å¨çº¿æµé(åºäºä¸é¢æ¬å°æµéæ建çæå¡,建议èªè¡æ建,ç®å大é¨åé½æäº)
-
å½å èªææå¡å¨Linuxæµéï¼ä½¿ç¨litespeed
# litespeedæµéï¼å»ºè®®å ³éç½éæµè¯ ./lite --config config.json --test çæé¾æ¥ # å¤å¶å°éænginxèµæºè·¯å¾ cp output.txt /usr/share/nginx/res/nodes.txt # base64ç¼ç ï¼çæv2ray订é base64 /usr/share/nginx/res/nodes.txt > /usr/share/nginx/res/node.txt
åèconfig.json é ç½®
{ "group": "Default", "speedtestMode": "pingonly", "pingMethod": "googleping", "sortMethod": "rspeed", "concurrency": 1024, "testMode": 2, "timeout": 5, "fontSize": 24, "outputMode": 4, "unique": true, "language": "en", "theme": "rainbow" }
èç¹è¿æ»¤
å é¤ä»¥ä¸å¯è½åå¨æµéé®é¢çèç¹
- SSR
- none
- rc4
- rc4-md5
- SS
- aes-128-cfb
- aes-256-cfb
- rc4-md5
- VMESS
- none
- grpc
- h2
- auto
项ç®çæå 容
èç¹
-
github action ( èç¹è¯¦æ )
- vless æªæµé (litespeedä¸æ¯æ)
- ååä¸è½¬æ¢(ä¸å«vless)
-
æ¬å°æ建 (github action èç¹æµè¯ä¸ºå½å¤æå¡å¨,å½å ä¸ä¿è¯è½ç¨,建议使ç¨æ¬å°äºæ¬¡æµéçéå使ç¨)
## windowsç³»ç»æ§è¡
localFilter.bat
## Linux /Mac OSç³»ç»æ§è¡
bash localFilter
## æè
chmod +x localFilter && ./localFilter
é»è®¤çæç为base64ç¼ç (v2rayN/ss/ssrç客æ·ç«¯å¯ç´æ¥ä½¿ç¨),å ¶ä»è¯·èªè¡ä½¿ç¨è®¢é 转æ¢è¿è¡è½¬æ¢
Hosts
走代çåipå¿åæ£æµ
- https://bgp.he.net/
- https://browserleaks.com/
- https://ip.voidsec.com/
- https://ipinfo.io/
- https://ipleak.com/
- https://ipleak.net/
- https://ipleak.org/
- https://ipx.ac/run
- https://nstool.netease.com/
- https://test-ipv6.com/
- https://whatismyipaddress.com/blacklist-check
- https://whoer.net/
- https://www.astrill.com/dns-leak-test
- https://www.astrill.com/ipv6-leak-test
- https://www.astrill.com/port-scan
- https://www.astrill.com/vpn-leak-test
- https://www.astrill.com/what-is-my-ip
- https://www.deviceinfo.me/
- https://www.dnsleaktest.com/
- https://www.doileak.com/
- https://www.expressvpn.com/webrtc-leak-test
使ç¨è½¯ä»¶
å¹³å° | 软件 | æ¯æåè®® |
---|---|---|
Windows | V2rayN | SSãTrojanãVmessãVLESS |
Windows | Clash CFW (æ¨è) å·²Gèªè¡æç´¢å®è£ å | SSãSSRãTrojanãVmessãVLESS |
macOS | ClashX å·²Gèªè¡æç´¢å®è£ å | SSãSSRãTrojanãV2ray |
macOS | V2rayU | SSãSSRãTrojanãV2ray |
Android | V2rayNG | SSãTrojanãV2rayï¼VmessãVLESSï¼ãXray |
Android | ClashForAndroid (æ¨è) å·²Gèªè¡æç´¢å®è£ å | SSãSSRãTrojanãVmessãVLESS |
Android | NekoBoxForAndroid (æ¨è) | VMess / VLESS / SSR / Trojan / Trojan-Go/ NaiveProxy / HTTP(S) / SOCKS5/etc. |
IOS | Shadowrocket å°ç«ç® IOSéå½åºè´ä¹° | SSãSSRãTrojanãV2rayãVLESS |
IOS | Quantumult IOSéå½åºè´ä¹° | SSãSSRãTrojanãV2ray |
IOS | QuantumultX IOSéå½åºè´ä¹° | SSãSSRãTrojanãV2ray |
Stargazers over time
声æ
æ¬é¡¹ç®ä» é个人èªå·±ä½¿ç¨ï¼ç¦æ¢ä½¿ç¨æ¬é¡¹ç®è¿è¡è¥å©ååå ¶ä»è¿æ³äºæ ï¼äº§ççä¸ååææ¬é¡¹ç®æ¦ä¸è´è´£
Top Related Projects
Utility to convert between various subscription format
SSR 去广告ACL规则/SS完整GFWList规则/Clash规则碎片,Telegram频道订阅地址
机场推 荐/SSR V2ray节点订阅机场/镜像直连/工具推荐
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot