I also have a paper dict 
but I can only bear using it if look-ups turn out to be up to one item per page, which is still a rare situation.
3 Likes
My answer is “it depends”, but for weird reasons. There are plenty of words I don’t know but don’t look up since they are obvious from context. But I also have a bad habit of second guessing myself, wanting to see if there’s specific nuance, or getting curious about the pitch accent, so I look up many words unnecessarily.
Yeah, I know. Just using myself as an example to say “not everyone does this” even if there are people who do.
Yeah, unfortunately there will always be niche books on a language learning site that get fewer ratings and are at greater risk of the problems you mention. For example, I read a lot of 4-koma manga and very often I’m the only one grading them, which isn’t super helpful.
I do find your commentary on videos particularly interesting as well. I probably have contributed to some of the problem there since I’ve watched several anime way above my level (with Japanese subs) and got by simply because I’d already watched them with English subs previously. For example, Vivy is L33, Saiki K is L32, Oshi no Ko is L30, and Violet Evergarden is L28, but I can’t say I really have a good sense of their relative difficulties from personal experience (and my real level for listening is probably closer to L25). (My gradings don’t seem wrong since it looks like I have apparently mostly graded consistently with the consensus, but I’m not sure I could describe to you why one is harder than another.)
5 Likes
I do slow digital look ups by using my digital dictionary (it’s just the ebook version of the dictionary, but the search bar means it’s faster to look things up) whenever I can.
However, when I am reading with the booklive app, I am just using the built in dictionary because getting out of the app would be annoying.
… that being said, that point is mostly moot since I look up 0~2 words per book…
7 Likes
That’s a both It depends.
For the first question, it will depend if I’m doing physical or digital reading, and the source.
For the second one, it will depend on the book.
2 Likes
Funny, I think I looked up 5-10 words when reading The Way of Kings in English a few weeks back. Which is a great example of why easy lookups can be counterproductive, because I didn’t need to look up the words so much as I got curious even though I would never have bothered if I was reading physically.
6 Likes
Gradings are subjective; there’s no escaping that. Even those who attend the same classes, use the same learning materials, and spend the same time studying will have differing opinions on how difficult something is. Natively uses the wisdom of the crowd to gauge difficulty, so the more people contribute, the more accurate the level. It’s not a rigorous academic study where you can control who reads what.
Who’s to say how difficult something “really” is? Everyone (including you) is judging by their own perception of how difficult it felt to them. You can take different aspects of the language into consideration, but this doesn’t negate the fact that it’s subjective.
Visual storytelling helps provide context, which naturally aids understanding; should those who read novels disregard the plot because the natural patterns found in narratives give contextual clues that help with understanding the language?
This is your opinion - there is no objective measurement of difficulty that makes you right and others wrong.
Are you proposing that only advanced learners be allowed to grade items? Then exceptionally few items suitable for beginners and intermediate would have gradings at all, not to mention you’d be excluding the people who could actually give feedback on whether or not something is suitable for the level that they are at.
You could make a whole list of things you think people should or shouldn’t take into account when grading, but it’s hardly as if you could enforce this. You could make a rule that only those who provide proof of having passed the JLPT N1 are allowed to grade items, but then there would be very few items graded, most of which would only be graded by one person. And how would you be able to tell if they’ve judged it well? How would you know they even read the book at all? I could go on, but you get the idea.
While I can agree on some of the issues you raise (I think Detailed language evaluations would be immensely useful for providing more information about what makes something difficult), I think a lot of what you’re unhappy with just stems from the subjectivity of grading, which is not something that’s going to change - everyone has different learning methods, experiences, and opinions, and this affects how they grade item difficulty.
Rather than being a flaw, I think it’s more beneficial to have a wide variety of experiences, especially if people include reviews - it makes the grading result more accurate and is helpful to more people. You could have an expert draft a list of media in order of difficulty according to their professional opinion, or have students taking the same class all read and grade the same items, or even create something similar to JPDB that takes more than just vocabulary into account… But none of these would be able to produce objective difficulty levels, either. And they certainly wouldn’t provide the volume of information that Natively does - over 12k Japanese books have non-temporary ratings.
One last thought: Natively is merely a helpful guide, it isn’t claiming to be a definitive source of the objective difficulty of foreign-language media. Difficulty levels fluctuate, and people can disagree with the level of even the most graded items, but that doesn’t detract from how useful it can be for finding things to enjoy at whatever level you are at in the language you are studying.
16 Likes
I’m not proposing anything. Realistically, there is no solution to any of these problems. The question is “Do you trust Natively levels” and this is just some food for thought about why it might not be quite as good as a lot of people here say. It isn’t a bad site or system, it just has some issues to be wary of.
The point wasn’t that visuals don’t help or are bad to consider, it’s that when one’s comprehension is significantly below the material, they can convince themselves that something was easier than some other thing just because it had cool pictures to follow along with. That they would have come to the same conclusion had there been no words at all. I’m tired of hearing people weeks into studying on discord telling other beginners that so-and-so anime is great for beginners or that this-and-that other series is bad for beginners when the person giving this advice does not understand enough of either show to make that judgment reliably.
I’m all for subjectivity, but let’s be honest, some gradings lower the quality of the results.
2 Likes
Great discussion here! I’ve always thought about the levels but was too lazy to make a thread about it.
As far as lookups go, I intentionally choose books where at most i’ll have to look up say 2 or 3 words per 100 words. I keep track of my known vocab which helps me choose the right books, but if there’s a book I want to read that requires more vocab, I usually just learn the extra vocab through SRS before I start reading.
We’re all different here but I personally really cannot tolerate looking up so many words while I read, it takes me out of the experience, I don’t want to feel like I’m learning a language while I read.
I havent graded anything in awhile, because I added all the anime I watched in one go, now I have like 500+ gradings and I just cant be bothered, nor can I remember the difficulty.
2 Likes
Conversely, some advanced learners suggest items for beginners because it seems easy to them now, rather than considering how difficult they found it at the time.
But what harm is done if someone watches or reads something above their level, whether it was suggested to them by someone else or not? If they enjoy it, great - if not, they can just choose something else. Better to have a lot of suggestions than none at all, even if they’re not perfect.
But it’s not as though Natively provides detailed rules for grading, it’s merely “what did you find more difficult?”, and all people can do is answer honestly based on their own knowledge and perceptions. As a learner, it’s natural there are things one doesn’t know (and doesn’t know they don’t know ) - I don’t see how that can affect the “quality” unless you’re only interested in the opinions of specific people (advanced learners, for example).
9 Likes
words → pages? There must be an error here, as this would mean 12-15 words per page. The rest does not sound like you could stand that, even though there are people that can.
2 Likes
I also feel like it’s great to have gradings from both beginners and more advanced learners. Everyone will notice different things, so that way we get an average level that encompasses as much as possible (from someone reading above their level that misses a lot of things but still manages to follow along, to someone for whom this is super easy to read). It also means that the levels will still make sense and be helpful if you read above/below your level.
Reading above your level, is, after all, an integral part of language learning. You have to do it at some point, and then that level that was once hard becomes easier. So it’s definitely valuable to have levels that take into account the experience of those that read above their level as well.
9 Likes
It definitely is. Reading your first proper no-pictures novel is like slamming up against a brick wall, followed by a 100 kilometer trudge through a tar pit. In fact, the next two or three are also pretty rough unless the content is on the easier side.
I mainly read on desktop using ttsu and yomichan, so lookups are near immediate. I tend to do very fast checks on new words to see the meaning and frequency, and then I will use the built-in “add to anki” feature if it’s within my target frequency range (or if it’s an obviously useful word).
When I’m in transit or out of the house I read on a kindle paperwhite, which has a built-in pop-up dictionary. It’s nice but since it’s an older e-ink device it’s much slower and clunkier, so I tend to try and avoid lookups unless I really don’t know it, and I’ll highlight the word.
I don’t read often on paper, but when I do I am almost guaranteed to know the kanji, so I’ll take a guess as to the reading, and if I get it wrong I’ll just write a word I do know with each character. I really hate trying to use OCR on my phone, it’s often slower for me than the way I do it currently, and same for those “construct it from the radicals” search boxes
8 Likes
Indeed. I think DuoUnderscore’s concerns about really low level users are legit, but the opposite is a problem too. Advanced readers can’t easily differentiate between levels of easiness and will often grade books that should be 5+ levels apart as being the same. That’s why it’s important to have gradings by a variety of users.
10 Likes
I’m curious: what are you looking to get out of Natively?
As an example, I love that I get exposed to books especially, but also other media that I wouldn’t know about otherwise, and that I have a vague guide as to the amount of effort that material would be be for me to read.
Additionally, book clubs have been really instrumental in me pushing my level as I apparently am motivated to read with a group, but that’s less about Natively specifically.
I really don’t want this to be a pile on thread bc I feel like that’s where we’re maybe going, but two things here:
-
Do you think you would have gotten to your current level having not read far above your level with tools?
-
What about the situation where you “know” all the words but don’t understand the sentence? That requires zero tools but I think we can agree that that would be above someone’s level.
6 Likes
I had hoped for a way to find content at a very similar level to other content I was very comfy with so I could gradually see the rise, and it’s close but a bit too hit-or-miss so now I just find longer shows/book series and err into too-easy stuff when I branch out.
Book clubs here look fun! Haven’t tried any yet, but they look pretty nice to participate in.
I do (and maybe would have been better off for it), but my stance on the levels thing is unrelated to learning strategy. If you want to read hard books, go for it, I’ll encourage it. I’m just acknowledging that tool assistance and community influence can upset the gradings.
I try my best to avoid this scenario; I aim for 100% of the sentences I read to be understood even if I don’t know all the words. I’m a very particular case in that regard, hence why I wish (and I seem to stand alone on this) that the levels would be pretty consistent, and that when I open a 26 I wouldn’t get something harder than other 28s and when I go to watch a 29 rated series, I don’t get something that’s probably more like a 24. Again, it’s sometimes pretty good but a lot of the time it’s just like… way off, and then you look at the written reviews and you can see why.
4 Likes
With the risk of making the system more complicated and not sure how useful in the end it would be:
Maybe aside from grading, having a way to vote if you agree/disagree/strongly disagree with a book level, as to gather data and further tweak the algorithm?
6 Likes
The thing is, gradings are always going to be somewhat subjective. If you compare a book with slightly more difficult grammar to one with slightly more difficult vocab, should they be the same level or should one be higher than the other, and which? There’s no right answer to things like this, it’s all personal perspectives. So something that might seem easier than something else to you might seem the opposite for someone else.
Another factor could be what type of vocab you’ve exposed yourself to. For example, I’ve started watching Single’s inferno | L25?? and while there’s technically a very small range of vocab, it’s also a lot of vocab that I’m not used to, so at times it feels harder to follow along than your regular romance drama to me, because I’ve watched a lot of those and specifically sat down to learn a lot of the vocab. (I’m also really bad at konglish, because, again, I’m just not used to hearing it) Someone that has watched mostly reality tv, on the other hand, would likewise struggle more with dramas.
9 Likes
Perhaps. It’s possible fodder for a request.
In all honesty though, I think the best way to improve the algorithm is to simply look at the grading data and see how self-consistent it is. Some books & tv shows are bound to have more variance for all the reasons @DuoUnderscore and others have stated and it’d be nice to surface that information. I can’t find the request, but I thought there was a feature request to surface historical rating level changes and compute how large the variant / how consistent something is.
I also agree with a lot of @DuoUnderscore says around Video especially. I think since a lot of books have a narrator that speaks in one style, that can make books feel a bit more consistent? Not true for manga of course. I also agree with @DuoUnderscore that people are watching stuff that’s above their level in Video more consistently, especially with tools like language reactor built into Netflix.
I really do like @bibliothecary 's Detailed language evaluations suggestion too. I think that’d illuminate the ‘why’ questions you’re asking on how difficulty is being assesed.
Ultimately though, it comes down:
- is it helpful for tracking your progress?
- is it helpful for discovering new content?
I think you’re right that it’s not 10/10 for each of those, especially for video. And there is a lot of work that can be done to analyze possible systematic biases. And I like to hear opinions from people who don’t find it useful and prefer not to use them. I must admit though, that while I do get frustrated it’s not totally accurate, even if flawed, I find it very nice.
Granted, I did start it, so hopefully I think that 
15 Likes
I’m not sure what error you’re talking about but referring to words per page wouldn’t work out since word count per page varies quite a bit.
If i’m looking up 2 or 3 words per 100 words, that means I should understand 97-98% of what I’m reading, if we can ignore writing style for a moment. So a very tolerable number in my opinion.
Sometime last year I remember coming across some research saying the 98%+ is the ideal for extensive reading.
I hope that made that sense
That’s why I only stay in that 97-98% range. I lack the perseverance that some people have on this forum (how do they do it?)
Found this as I was writing this post:
3 Likes
But on the other hand you voted above that you “hardly ever“ make look-ups.
Sorry, there is probably some misunderstanding on my side: you probably count 2-3% of the words as hardly ever, while for me, when I look up 2-3% of the words which are 12-15 words/page in a standard novel, that is identical to looking up every new word (in a mid level 35 novel). For myself I think that this is a hard thing to do, but for you - with a pop-up dict - this probably does not feel that hard.
2 Likes
